[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:34 dist-test-slave-m4lt krb5kdc[10193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:34 dist-test-slave-m4lt krb5kdc[10193](info): set up 2 sockets
Sep 27 15:16:34 dist-test-slave-m4lt krb5kdc[10193](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 15:16:35.074489 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:35 dist-test-slave-m4lt krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:35.190397 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:38127
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:35.297678 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:35.297936 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:35.297976 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:35.301944 10206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:35.302002 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:35.302019 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:35.302035 10206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:35.302052 10206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:35.307384 10206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:38127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:35.308529 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:35.309950 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:35.319697 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:35.319751 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:35.319933 10206 server_base.cc:1034] running on GCE node
W20260927 15:16:35.319723 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:35.320616 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:35.321905 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:35.323091 10206 hybrid_clock.cc:648] HybridClock initialized: now 1790522195323060 us; error 37 us; skew 500 ppm
Sep 27 15:16:35 dist-test-slave-m4lt krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522195, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:35.326007 10206 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:35.326982 10206 webserver.cc:460] Webserver started at http://127.9.242.190:35391/ using document root <none> and password file <none>
I20260927 15:16:35.327529 10206 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:35.327600 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:35.327813 10206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:35.329909 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "8a43c8efe3e147a6b93d94b662b4ad7d"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "f4b3a41a7a8d0755f8afb00b36194f40"
server_key_iv: "9bc8e3b879369019dacab7e7611ac3c3"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.330436 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "8a43c8efe3e147a6b93d94b662b4ad7d"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "f4b3a41a7a8d0755f8afb00b36194f40"
server_key_iv: "9bc8e3b879369019dacab7e7611ac3c3"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.334149 10206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:35.336527 10221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:35.337757 10206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:35.337869 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "8a43c8efe3e147a6b93d94b662b4ad7d"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "f4b3a41a7a8d0755f8afb00b36194f40"
server_key_iv: "9bc8e3b879369019dacab7e7611ac3c3"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.337965 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:35.360954 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:35.364410 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:35.364594 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:35.371227 10206 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:38127
I20260927 15:16:35.371258 10273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:38127 every 8 connection(s)
I20260927 15:16:35.372153 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:35.375007 10274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:35.376583 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 10206
I20260927 15:16:35.376706 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:35.378072 10186 external_mini_cluster.cc:1428] Setting key de998e3050a72d7fd2859a211c33656a
I20260927 15:16:35.380728 10274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d: Bootstrap starting.
I20260927 15:16:35.383320 10274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:35.384018 10274 log.cc:824] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:35.385898 10274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d: No bootstrap required, opened a new log
I20260927 15:16:35.389262 10274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38127 } }
I20260927 15:16:35.389487 10274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:35.389554 10274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a43c8efe3e147a6b93d94b662b4ad7d, State: Initialized, Role: FOLLOWER
I20260927 15:16:35.390027 10274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38127 } }
I20260927 15:16:35.390172 10274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:35.390244 10274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:35.390336 10274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:35.391258 10274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38127 } }
I20260927 15:16:35.391589 10274 leader_election.cc:302] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a43c8efe3e147a6b93d94b662b4ad7d; no voters:
I20260927 15:16:35.391887 10274 leader_election.cc:288] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:35.391968 10279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:35.392127 10279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [term 1 LEADER]: Becoming Leader. State: Replica: 8a43c8efe3e147a6b93d94b662b4ad7d, State: Running, Role: LEADER
I20260927 15:16:35.392493 10279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38127 } }
I20260927 15:16:35.393152 10274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:35.394088 10281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a43c8efe3e147a6b93d94b662b4ad7d. Latest consensus state: current_term: 1 leader_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38127 } } }
I20260927 15:16:35.394208 10281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:35.394500 10280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a43c8efe3e147a6b93d94b662b4ad7d" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38127 } } }
I20260927 15:16:35.394586 10280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:35.394585 10288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 15:16:35 dist-test-slave-m4lt krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:35.397984 10288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:35.401640 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.381103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.9.242.190:38127)
0927 15:16:35.381554 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:35.381568 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:35.382564 (+ 996us) server_negotiation.cc:407] Connection header received
0927 15:16:35.385615 (+ 3051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:35.385631 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:35.386018 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:35.386325 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:35.388426 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.389385 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.390738 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.391191 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.391342 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:35.395981 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:35.395987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:35.396002 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:35.396032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:35.397988 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:35.398604 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:35.398610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:35.398615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:35.398745 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:35.399130 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:35.399133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:35.399135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:35.399604 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:35.399779 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:35.400463 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 15:16:35.400751 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":120,"threads_started":1}
I20260927 15:16:35.403677 10288 catalog_manager.cc:1347] Generated new cluster ID: 38df21eadfe946be918c07c6b04819e2
I20260927 15:16:35.403740 10288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:35.421499 10288 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:35.422766 10288 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:35.429145 10288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a43c8efe3e147a6b93d94b662b4ad7d: Generated new TSK 0
I20260927 15:16:35.429993 10288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:35.458066 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38127
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:35.565630 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:35.565924 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:35.565969 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:35.569895 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:35.569952 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:35.570010 10300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:35.575139 10300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:35.576121 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:35.577039 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:16:35.586403 10300 server_base.cc:1034] running on GCE node
W20260927 15:16:35.586354 10308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:35.586357 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:35.586369 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:35.586992 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:35.587608 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:35.588768 10300 hybrid_clock.cc:648] HybridClock initialized: now 1790522195588758 us; error 41 us; skew 500 ppm
Sep 27 15:16:35 dist-test-slave-m4lt krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522195, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:35.596343 10300 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:35.597223 10300 webserver.cc:460] Webserver started at http://127.9.242.129:40627/ using document root <none> and password file <none>
I20260927 15:16:35.597819 10300 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:35.597888 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:35.598089 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:35.600106 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "dc7b64cf03e74a86aded39f07aa73707"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "d18e936edf578344eb3eb857c9e259da"
server_key_iv: "35b44e9f018fd4086ad05170c1c222b5"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.600646 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "dc7b64cf03e74a86aded39f07aa73707"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "d18e936edf578344eb3eb857c9e259da"
server_key_iv: "35b44e9f018fd4086ad05170c1c222b5"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.604425 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:16:35.606845 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:35.608013 10300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:35.608129 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "dc7b64cf03e74a86aded39f07aa73707"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "d18e936edf578344eb3eb857c9e259da"
server_key_iv: "35b44e9f018fd4086ad05170c1c222b5"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.608224 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:35.636610 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:35.639353 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:35.639521 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:35.640079 10300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:35.640908 10300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:35.640955 10300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:35.641006 10300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:35.641032 10300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:35.649724 10300 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:35629
I20260927 15:16:35.649809 10428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:35629 every 8 connection(s)
I20260927 15:16:35.650612 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:35.653638 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 10300
I20260927 15:16:35.653752 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:35.653985 10186 external_mini_cluster.cc:1428] Setting key fba4b944f57da96ec114927de3c873f0
Sep 27 15:16:35 dist-test-slave-m4lt krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522195, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:16:35.664191 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.652270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:52275 (local address 127.9.242.190:38127)
0927 15:16:35.652389 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:35.652393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:35.653236 (+ 843us) server_negotiation.cc:407] Connection header received
0927 15:16:35.654546 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:35.654549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:35.654602 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:35.654679 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:35.656081 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.656603 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.657804 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.658137 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.658239 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:35.660737 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:35.660741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:35.660744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:35.660775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:35.662309 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:35.662782 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:35.662786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:35.662788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:35.662882 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:35.663200 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:35.663202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:35.663203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:35.663388 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:35.663462 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:35.663938 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 15:16:35.664063 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:16:35.665048 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.652662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38127 (local address 127.9.242.129:52275)
0927 15:16:35.653090 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:35.653132 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:35.654348 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:35.654846 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:35.654853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:35.655273 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:35.655934 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:35.655955 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.656731 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.656734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:35.657695 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:35.657705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.658273 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.658276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:35.658423 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:35.658995 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:35.659011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:35.660561 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:35.662421 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:35.662426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:35.662439 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:35.662691 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:35.662968 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:35.662970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:35.662972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:35.663111 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:35.663463 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:35.663469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:35.663714 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 15:16:35.663877 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:16:35.664161 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:16:35.666373 10429 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38127
I20260927 15:16:35.666613 10429 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:35.667096 10429 heartbeater.cc:511] Master 127.9.242.190:38127 requested a full tablet report, sending...
I20260927 15:16:35.668627 10238 ts_manager.cc:194] Registered new tserver with Master: dc7b64cf03e74a86aded39f07aa73707 (127.9.242.129:35629)
I20260927 15:16:35.669806 10238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:52275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:35.678305 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38127
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:35.788197 10434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:35.788456 10434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:35.788499 10434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:35.792516 10434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:35.792582 10434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:35.792663 10434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:35.798000 10434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:35.799037 10434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:35.800019 10434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:35.809139 10439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:35.809247 10434 server_base.cc:1034] running on GCE node
W20260927 15:16:35.809151 10442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:35.809438 10440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:35.809831 10434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:35.810474 10434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:35.811625 10434 hybrid_clock.cc:648] HybridClock initialized: now 1790522195811602 us; error 37 us; skew 500 ppm
Sep 27 15:16:35 dist-test-slave-m4lt krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522195, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:35.814399 10434 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:35.815214 10434 webserver.cc:460] Webserver started at http://127.9.242.130:32873/ using document root <none> and password file <none>
I20260927 15:16:35.815721 10434 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:35.815780 10434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:35.815963 10434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:35.817936 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "1ff53f8ff0c340e29be25116dce8a965"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "5fb646c845cd80883d376da8678d288f"
server_key_iv: "d6f1a2e6772ad498ac87b15fb74f3d9a"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.818475 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "1ff53f8ff0c340e29be25116dce8a965"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "5fb646c845cd80883d376da8678d288f"
server_key_iv: "d6f1a2e6772ad498ac87b15fb74f3d9a"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.822023 10434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 15:16:35.824509 10449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:35.825834 10434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:35.825942 10434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "1ff53f8ff0c340e29be25116dce8a965"
format_stamp: "Formatted at 2026-09-27 15:16:35 on dist-test-slave-m4lt"
server_key: "5fb646c845cd80883d376da8678d288f"
server_key_iv: "d6f1a2e6772ad498ac87b15fb74f3d9a"
server_key_version: "encryptionkey@0"
I20260927 15:16:35.826047 10434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:35.850243 10434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:35.853193 10434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:35.853397 10434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:35.854018 10434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:35.854851 10434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:35.854897 10434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:35.854950 10434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:35.854978 10434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:35.863503 10434 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:34021
I20260927 15:16:35.863524 10562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:34021 every 8 connection(s)
I20260927 15:16:35.864420 10434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
Sep 27 15:16:35 dist-test-slave-m4lt krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522195, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:35.873915 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 10434
I20260927 15:16:35.874029 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:35.874286 10186 external_mini_cluster.cc:1428] Setting key 759c6ce26fe7aaa2171d47824da702a5
I20260927 15:16:35.877794 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.866160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:59015 (local address 127.9.242.190:38127)
0927 15:16:35.866300 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:35.866305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:35.867175 (+ 870us) server_negotiation.cc:407] Connection header received
0927 15:16:35.868410 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:35.868413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:35.868459 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:35.868535 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:35.869758 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.870340 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.871433 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.871653 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.871737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:35.874073 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:35.874077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:35.874080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:35.874109 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:35.875583 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:35.876146 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:35.876148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:35.876150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:35.876222 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:35.876579 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:35.876581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:35.876582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:35.876735 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:35.876820 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:35.877481 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 15:16:35.877590 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:16:35.878739 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:35.866565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38127 (local address 127.9.242.130:59015)
0927 15:16:35.867033 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:35.867075 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:35.868185 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:35.868704 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:35.868711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:35.869077 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:35.869590 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:35.869604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.870438 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.870440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:35.871319 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:35.871327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:35.871768 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:35.871771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:35.871920 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:35.872442 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:35.872457 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:35.873895 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:35.875693 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:35.875699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:35.875714 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:35.876042 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:35.876315 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:35.876318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:35.876320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:35.876497 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:35.876833 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:35.876840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:35.877149 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 15:16:35.877469 (+ 320us) client_negotiation.cc:240] Negotiation successful
0927 15:16:35.877918 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
I20260927 15:16:35.879834 10563 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38127
I20260927 15:16:35.880055 10563 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:35.880549 10563 heartbeater.cc:511] Master 127.9.242.190:38127 requested a full tablet report, sending...
I20260927 15:16:35.881543 10238 ts_manager.cc:194] Registered new tserver with Master: 1ff53f8ff0c340e29be25116dce8a965 (127.9.242.130:34021)
I20260927 15:16:35.882092 10238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:59015
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:35.901069 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38127
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:36.011291 10568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:36.011571 10568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:36.011634 10568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:36.015784 10568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:36.015852 10568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:36.015936 10568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:36.021291 10568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38127
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:36.022387 10568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:36.023362 10568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:36.032541 10576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:36.032591 10573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:36.032562 10568 server_base.cc:1034] running on GCE node
W20260927 15:16:36.032721 10574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:36.033139 10568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:36.033823 10568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:36.034997 10568 hybrid_clock.cc:648] HybridClock initialized: now 1790522196034937 us; error 78 us; skew 500 ppm
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522196, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:36.037549 10568 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:36.038399 10568 webserver.cc:460] Webserver started at http://127.9.242.131:36743/ using document root <none> and password file <none>
I20260927 15:16:36.038913 10568 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:36.038975 10568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:36.039150 10568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:36.041098 10568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "6e58e8058e7543e1a0d46a7697e22246"
format_stamp: "Formatted at 2026-09-27 15:16:36 on dist-test-slave-m4lt"
server_key: "632b45f12624d9c63a5157d426e6b2e7"
server_key_iv: "64f78c3e124d505efd1e969515f6dfb7"
server_key_version: "encryptionkey@0"
I20260927 15:16:36.041620 10568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "6e58e8058e7543e1a0d46a7697e22246"
format_stamp: "Formatted at 2026-09-27 15:16:36 on dist-test-slave-m4lt"
server_key: "632b45f12624d9c63a5157d426e6b2e7"
server_key_iv: "64f78c3e124d505efd1e969515f6dfb7"
server_key_version: "encryptionkey@0"
I20260927 15:16:36.045090 10568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:36.047446 10583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:36.048620 10568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:16:36.048722 10568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "6e58e8058e7543e1a0d46a7697e22246"
format_stamp: "Formatted at 2026-09-27 15:16:36 on dist-test-slave-m4lt"
server_key: "632b45f12624d9c63a5157d426e6b2e7"
server_key_iv: "64f78c3e124d505efd1e969515f6dfb7"
server_key_version: "encryptionkey@0"
I20260927 15:16:36.048823 10568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:36.059141 10568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:36.062063 10568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:36.062227 10568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:36.062758 10568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:36.063546 10568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:36.063591 10568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:36.063642 10568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:36.063670 10568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:36.072573 10568 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:44625
I20260927 15:16:36.072602 10696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:44625 every 8 connection(s)
I20260927 15:16:36.073411 10568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:36.076566 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 10568
I20260927 15:16:36.076676 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:36.076920 10186 external_mini_cluster.cc:1428] Setting key 49016fdb0c0ef3ec107b7dfe0ccc98cd
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522196, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:36.086402 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:36.075056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:38975 (local address 127.9.242.190:38127)
0927 15:16:36.075171 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:36.075175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:36.075977 (+ 802us) server_negotiation.cc:407] Connection header received
0927 15:16:36.077522 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:36.077525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:36.077583 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:36.077682 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:36.078983 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.079567 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.080737 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.080947 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.081030 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:36.083238 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:36.083241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:36.083243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:36.083272 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:36.084544 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:36.084999 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:36.085003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:36.085004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:36.085074 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:36.085367 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:36.085370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:36.085372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:36.085537 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:36.085616 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:36.086120 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 15:16:36.086230 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:16:36.087162 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:36.075403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38127 (local address 127.9.242.131:38975)
0927 15:16:36.075832 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:36.075873 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:36.077300 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:36.077863 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:36.077870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:36.078373 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:36.078855 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:36.078868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.079682 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.079685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:36.080629 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:36.080638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.081075 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.081078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:36.081234 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:36.081769 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:36.081784 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:36.083094 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:36.084646 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:36.084652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:36.084668 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:36.084912 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:36.085150 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:36.085153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:36.085156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:36.085287 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:36.085626 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:36.085633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:36.085906 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 15:16:36.086067 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:16:36.086376 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260927 15:16:36.088160 10697 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38127
I20260927 15:16:36.088366 10697 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:36.088790 10697 heartbeater.cc:511] Master 127.9.242.190:38127 requested a full tablet report, sending...
I20260927 15:16:36.089728 10238 ts_manager.cc:194] Registered new tserver with Master: 6e58e8058e7543e1a0d46a7697e22246 (127.9.242.131:44625)
I20260927 15:16:36.090402 10238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:38975
I20260927 15:16:36.091059 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:16:36.116621 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:36.107901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.9.242.129:35629)
0927 15:16:36.108187 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:36.108201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:36.108225 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:16:36.108331 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:36.108337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:36.108533 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:36.108672 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:36.109263 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.110093 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.110883 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.111263 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.111347 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:36.113710 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:36.113721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:36.113734 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:36.113759 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:36.115140 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:36.115543 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:36.115548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:36.115549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:36.115626 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:36.115827 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:36.115829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:36.115830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:36.116047 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:36.116210 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:36.116351 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:16:36.116472 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
W20260927 15:16:36.117589 10343 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57478
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:16:36.141330 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:36.133617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.9.242.129:35629)
0927 15:16:36.133811 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:36.133815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:36.133870 (+ 55us) server_negotiation.cc:407] Connection header received
0927 15:16:36.133992 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:36.133995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:36.134036 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:36.134094 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:36.134826 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.135321 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.136115 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:36.136325 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:36.136390 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:36.138607 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:36.138610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:36.138613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:36.138636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:36.139869 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:36.140302 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:36.140304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:36.140306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:36.140374 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:36.140562 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:36.140564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:36.140565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:36.140741 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:36.140854 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:36.141111 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 15:16:36.141207 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:16:36.142290 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 10300
I20260927 15:16:36.148124 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 10434
I20260927 15:16:36.153220 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 10568
I20260927 15:16:36.157348 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 10206
2026-09-27T15:16:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2058 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10711](info): set up 2 sockets
Sep 27 15:16:36 dist-test-slave-m4lt krb5kdc[10711](info): commencing operation
krb5kdc: starting...
W20260927 15:16:37.109180 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:37 dist-test-slave-m4lt krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:37.224803 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34381
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:37.331568 10724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:37.331853 10724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:37.331910 10724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:37.335975 10724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:37.336035 10724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:37.336061 10724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:37.336081 10724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:37.336112 10724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:37.341325 10724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:34381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:37.342383 10724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:37.343411 10724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:37.351418 10730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:37.351430 10732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:37.351502 10729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:37.351624 10724 server_base.cc:1034] running on GCE node
I20260927 15:16:37.352218 10724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:37.353320 10724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:37.354480 10724 hybrid_clock.cc:648] HybridClock initialized: now 1790522197354452 us; error 41 us; skew 500 ppm
Sep 27 15:16:37 dist-test-slave-m4lt krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:37.357004 10724 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:37.357851 10724 webserver.cc:460] Webserver started at http://127.9.242.190:34773/ using document root <none> and password file <none>
I20260927 15:16:37.358392 10724 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:37.358458 10724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:37.358628 10724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:37.360551 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "84f86a2405aa45a78bfade56e0e811c4"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "9605ece8af91faf2a573afb9f11663ce"
server_key_iv: "255294931fd2cceb55619fe913bdf55b"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.361064 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "84f86a2405aa45a78bfade56e0e811c4"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "9605ece8af91faf2a573afb9f11663ce"
server_key_iv: "255294931fd2cceb55619fe913bdf55b"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.364620 10724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 15:16:37.366978 10739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:37.368135 10724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:37.368245 10724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "84f86a2405aa45a78bfade56e0e811c4"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "9605ece8af91faf2a573afb9f11663ce"
server_key_iv: "255294931fd2cceb55619fe913bdf55b"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.368366 10724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:37.387455 10724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:37.391070 10724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:37.391288 10724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:37.398524 10724 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:34381
I20260927 15:16:37.398545 10791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:34381 every 8 connection(s)
I20260927 15:16:37.399513 10724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:37.400658 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 10724
I20260927 15:16:37.400758 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:37.401059 10186 external_mini_cluster.cc:1428] Setting key bc2fc6c285bbd0d88f598593db3c49e4
I20260927 15:16:37.402884 10792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:37.409387 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4: Bootstrap starting.
Sep 27 15:16:37 dist-test-slave-m4lt krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:37.411736 10792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:37.412438 10792 log.cc:824] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:37.414346 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4: No bootstrap required, opened a new log
I20260927 15:16:37.414732 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.402482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.9.242.190:34381)
0927 15:16:37.402863 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:37.402878 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:37.402912 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:16:37.403558 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:37.403575 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:37.403923 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:37.404229 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:37.405038 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.406061 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.407187 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.407662 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.407787 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:37.410452 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:37.410458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:37.410471 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:37.410499 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:37.412071 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:37.412499 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:37.412504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:37.412508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:37.412636 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:37.412876 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:37.412879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:37.412881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:37.413263 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:37.413427 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:37.413729 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 15:16:37.413995 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260927 15:16:37.418196 10792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f86a2405aa45a78bfade56e0e811c4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34381 } }
I20260927 15:16:37.418481 10792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:37.418550 10792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f86a2405aa45a78bfade56e0e811c4, State: Initialized, Role: FOLLOWER
I20260927 15:16:37.418985 10792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f86a2405aa45a78bfade56e0e811c4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34381 } }
I20260927 15:16:37.419111 10792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:37.419157 10792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:37.419226 10792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:37.420046 10792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f86a2405aa45a78bfade56e0e811c4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34381 } }
I20260927 15:16:37.420339 10792 leader_election.cc:302] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f86a2405aa45a78bfade56e0e811c4; no voters:
I20260927 15:16:37.420588 10792 leader_election.cc:288] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:37.420702 10797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:37.420912 10797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [term 1 LEADER]: Becoming Leader. State: Replica: 84f86a2405aa45a78bfade56e0e811c4, State: Running, Role: LEADER
I20260927 15:16:37.421226 10797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f86a2405aa45a78bfade56e0e811c4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34381 } }
I20260927 15:16:37.421748 10792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:37.422639 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84f86a2405aa45a78bfade56e0e811c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f86a2405aa45a78bfade56e0e811c4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34381 } } }
I20260927 15:16:37.422746 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:37.423115 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f86a2405aa45a78bfade56e0e811c4. Latest consensus state: current_term: 1 leader_uuid: "84f86a2405aa45a78bfade56e0e811c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f86a2405aa45a78bfade56e0e811c4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34381 } } }
I20260927 15:16:37.423243 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:37.423847 10806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:37.426453 10806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:37.432010 10806 catalog_manager.cc:1347] Generated new cluster ID: e91e96538e494399afc1668734e79ea5
I20260927 15:16:37.432082 10806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:37.443100 10806 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:37.444033 10806 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:37.448683 10806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84f86a2405aa45a78bfade56e0e811c4: Generated new TSK 0
I20260927 15:16:37.449338 10806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:37.478276 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34381
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:37.584975 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:37.585253 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:37.585312 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:37.589282 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:37.589340 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:37.589421 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:37.594556 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:37.595583 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:37.596573 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:37.605792 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:37.605830 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:37.605792 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:37.606204 10818 server_base.cc:1034] running on GCE node
I20260927 15:16:37.606592 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:37.607203 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:37.608362 10818 hybrid_clock.cc:648] HybridClock initialized: now 1790522197608348 us; error 27 us; skew 500 ppm
Sep 27 15:16:37 dist-test-slave-m4lt krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:37.611083 10818 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:37.611943 10818 webserver.cc:460] Webserver started at http://127.9.242.129:46267/ using document root <none> and password file <none>
I20260927 15:16:37.612465 10818 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:37.612528 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:37.612708 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:37.614784 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "db269fa28c664a2faafff61744b3a483"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "6df821b89f3e5bab57b310d56403f770"
server_key_iv: "8c8e8be39b184e6a439369a5154989f4"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.615309 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "db269fa28c664a2faafff61744b3a483"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "6df821b89f3e5bab57b310d56403f770"
server_key_iv: "8c8e8be39b184e6a439369a5154989f4"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.618759 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:16:37.620970 10833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:37.622174 10818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:37.622275 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "db269fa28c664a2faafff61744b3a483"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "6df821b89f3e5bab57b310d56403f770"
server_key_iv: "8c8e8be39b184e6a439369a5154989f4"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.622364 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:37.636857 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:37.639827 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:37.639993 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:37.640556 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:37.641353 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:37.641400 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:37.641451 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:37.641479 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:37.650002 10818 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:34383
I20260927 15:16:37.650025 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:34383 every 8 connection(s)
I20260927 15:16:37.650885 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:37.654093 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 10818
I20260927 15:16:37.654184 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:37.654413 10186 external_mini_cluster.cc:1428] Setting key 47d20b92b51471817d993aff4e29dd5a
Sep 27 15:16:37 dist-test-slave-m4lt krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:16:37.664752 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.652613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:51399 (local address 127.9.242.190:34381)
0927 15:16:37.652726 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:37.652729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:37.653521 (+ 792us) server_negotiation.cc:407] Connection header received
0927 15:16:37.654823 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:37.654825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:37.654877 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:37.654964 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:37.656300 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.657053 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.658413 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.658709 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.658808 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:37.661253 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:37.661258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:37.661261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:37.661289 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:37.662847 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:37.663318 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:37.663321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:37.663324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:37.663411 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:37.663703 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:37.663706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:37.663708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:37.663934 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:37.664034 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:37.664518 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 15:16:37.664625 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:16:37.665589 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.652960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34381 (local address 127.9.242.129:51399)
0927 15:16:37.653373 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:37.653415 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:37.654602 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:37.655172 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:37.655180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:37.655637 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:37.656157 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:37.656171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.657157 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.657160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:37.658292 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:37.658303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.658859 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.658862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:37.659020 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:37.659562 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:37.659577 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:37.661086 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:37.662966 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:37.662971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:37.662983 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:37.663220 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:37.663498 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:37.663500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:37.663502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:37.663625 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:37.664022 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:37.664027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:37.664299 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 15:16:37.664457 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:16:37.664710 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:16:37.666920 10947 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34381
I20260927 15:16:37.667150 10947 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:37.667614 10947 heartbeater.cc:511] Master 127.9.242.190:34381 requested a full tablet report, sending...
I20260927 15:16:37.669138 10756 ts_manager.cc:194] Registered new tserver with Master: db269fa28c664a2faafff61744b3a483 (127.9.242.129:34383)
I20260927 15:16:37.670279 10756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:51399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:37.681480 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34381
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:37.789265 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:37.789532 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:37.789593 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:37.793546 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:37.793623 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:37.793725 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:37.798947 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:37.799948 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:37.800956 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:37.810307 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:37.810441 10952 server_base.cc:1034] running on GCE node
W20260927 15:16:37.810352 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:37.810341 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:37.810926 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:37.811549 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:37.812695 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790522197812678 us; error 30 us; skew 500 ppm
Sep 27 15:16:37 dist-test-slave-m4lt krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:37.815239 10952 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:37.816032 10952 webserver.cc:460] Webserver started at http://127.9.242.130:38155/ using document root <none> and password file <none>
I20260927 15:16:37.816569 10952 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:37.816628 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:37.816784 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:37.818846 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "5ed9a297025a40d9a83a4fadd31af16f"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "427fa3aab8227dc476aa3ac0043d0d58"
server_key_iv: "16fbf5b6bd0e948886d3a65333af5696"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.819455 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "5ed9a297025a40d9a83a4fadd31af16f"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "427fa3aab8227dc476aa3ac0043d0d58"
server_key_iv: "16fbf5b6bd0e948886d3a65333af5696"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.823201 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:16:37.825749 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:37.827122 10952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:37.827250 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "5ed9a297025a40d9a83a4fadd31af16f"
format_stamp: "Formatted at 2026-09-27 15:16:37 on dist-test-slave-m4lt"
server_key: "427fa3aab8227dc476aa3ac0043d0d58"
server_key_iv: "16fbf5b6bd0e948886d3a65333af5696"
server_key_version: "encryptionkey@0"
I20260927 15:16:37.827366 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:37.846305 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:37.849391 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:37.849567 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:37.850173 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:37.850973 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:37.851025 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:37.851078 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:37.851106 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:37.859375 10952 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:42163
I20260927 15:16:37.859407 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:42163 every 8 connection(s)
I20260927 15:16:37.860234 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:37.867038 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 10952
I20260927 15:16:37.867154 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:37.867439 10186 external_mini_cluster.cc:1428] Setting key 68558980920857ee5c8010ea2e172772
Sep 27 15:16:37 dist-test-slave-m4lt krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:37.873131 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.861907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:42359 (local address 127.9.242.190:34381)
0927 15:16:37.862055 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:37.862058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:37.862858 (+ 800us) server_negotiation.cc:407] Connection header received
0927 15:16:37.864106 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:37.864109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:37.864164 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:37.864237 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:37.865385 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.865971 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.867276 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.867502 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.867599 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:37.870024 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:37.870027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:37.870029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:37.870050 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:37.871232 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:37.871714 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:37.871718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:37.871719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:37.871790 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:37.872090 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:37.872093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:37.872095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:37.872301 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:37.872383 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:37.872869 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:16:37.872977 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:16:37.873975 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:37.862291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34381 (local address 127.9.242.130:42359)
0927 15:16:37.862706 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:37.862748 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:37.863879 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:37.864422 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:37.864429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:37.864768 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:37.865236 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:37.865249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.866082 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.866085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:37.867157 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:37.867169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:37.867653 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:37.867658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:37.867843 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:37.868387 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:37.868402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:37.869881 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:37.871341 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:37.871346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:37.871361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:37.871630 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:37.871867 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:37.871870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:37.871871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:37.872012 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:37.872392 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:37.872398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:37.872655 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 15:16:37.872817 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 15:16:37.873114 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20260927 15:16:37.875119 11081 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34381
I20260927 15:16:37.875332 11081 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:16:37.875800 11081 heartbeater.cc:511] Master 127.9.242.190:34381 requested a full tablet report, sending...
I20260927 15:16:37.876809 10756 ts_manager.cc:194] Registered new tserver with Master: 5ed9a297025a40d9a83a4fadd31af16f (127.9.242.130:42163)
I20260927 15:16:37.877359 10756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:42359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:37.890218 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34381
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:37.997964 11086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:37.998246 11086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:37.998289 11086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:38.002363 11086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:38.002430 11086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:38.002490 11086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:38.007864 11086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:38.008949 11086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:38.009995 11086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:38.019539 11094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:38.019546 11091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:38.019539 11092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:38.019600 11086 server_base.cc:1034] running on GCE node
I20260927 15:16:38.020162 11086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:38.020809 11086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:38.021960 11086 hybrid_clock.cc:648] HybridClock initialized: now 1790522198021943 us; error 33 us; skew 500 ppm
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:38.024642 11086 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:38.025560 11086 webserver.cc:460] Webserver started at http://127.9.242.131:40533/ using document root <none> and password file <none>
I20260927 15:16:38.026131 11086 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:38.026186 11086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:38.026346 11086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:38.028561 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "aad1e85abdc844238ea28e98ac29bbbc"
format_stamp: "Formatted at 2026-09-27 15:16:38 on dist-test-slave-m4lt"
server_key: "f80451360cbfe78de6d98be3552f72b4"
server_key_iv: "4acd591a9b4e01b4fe60240576b803e4"
server_key_version: "encryptionkey@0"
I20260927 15:16:38.029093 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "aad1e85abdc844238ea28e98ac29bbbc"
format_stamp: "Formatted at 2026-09-27 15:16:38 on dist-test-slave-m4lt"
server_key: "f80451360cbfe78de6d98be3552f72b4"
server_key_iv: "4acd591a9b4e01b4fe60240576b803e4"
server_key_version: "encryptionkey@0"
I20260927 15:16:38.032935 11086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:16:38.035279 11101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:38.036460 11086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:38.036557 11086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "aad1e85abdc844238ea28e98ac29bbbc"
format_stamp: "Formatted at 2026-09-27 15:16:38 on dist-test-slave-m4lt"
server_key: "f80451360cbfe78de6d98be3552f72b4"
server_key_iv: "4acd591a9b4e01b4fe60240576b803e4"
server_key_version: "encryptionkey@0"
I20260927 15:16:38.036643 11086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:38.052186 11086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:38.054878 11086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:38.055022 11086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:38.055529 11086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:38.056269 11086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:38.056313 11086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:38.056351 11086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:38.056370 11086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:38.064095 11086 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:44341
I20260927 15:16:38.064126 11214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:44341 every 8 connection(s)
I20260927 15:16:38.064893 11086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:38.065508 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 11086
I20260927 15:16:38.065603 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:38.065867 10186 external_mini_cluster.cc:1428] Setting key d22e7b1c2695cda7ccf3a1c97f05589e
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522198, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:38.077358 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.066716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:42695 (local address 127.9.242.190:34381)
0927 15:16:38.066859 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.066863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.067829 (+ 966us) server_negotiation.cc:407] Connection header received
0927 15:16:38.069064 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.069067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.069109 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.069169 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:38.070232 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.070737 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.071881 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.072110 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.072189 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.074486 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:38.074489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:38.074491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:38.074512 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:38.075775 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.076162 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.076164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.076165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.076229 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.076475 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.076477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.076478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.076626 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:38.076691 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.077142 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.077233 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:38.078171 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.067143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34381 (local address 127.9.242.131:42695)
0927 15:16:38.067687 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.067743 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:38.068889 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:38.069306 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:38.069313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:38.069644 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:38.070119 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.070132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.070854 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.070856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.071765 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.071774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.072235 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.072238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.072394 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.072872 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:38.072888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:38.074342 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:38.075855 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:38.075859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:38.075870 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:38.076094 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:38.076292 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:38.076294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:38.076295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:38.076414 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:38.076690 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:38.076694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:38.076922 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 15:16:38.077084 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 15:16:38.077319 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":108,"threads_started":1}
I20260927 15:16:38.079365 11215 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34381
I20260927 15:16:38.079607 11215 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:38.080076 11215 heartbeater.cc:511] Master 127.9.242.190:34381 requested a full tablet report, sending...
I20260927 15:16:38.080952 10756 ts_manager.cc:194] Registered new tserver with Master: aad1e85abdc844238ea28e98ac29bbbc (127.9.242.131:44341)
I20260927 15:16:38.081475 10756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:42695
I20260927 15:16:38.091253 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:38.100802 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.094480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.9.242.190:34381)
0927 15:16:38.094604 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.094608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.094758 (+ 150us) server_negotiation.cc:407] Connection header received
0927 15:16:38.094859 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.094861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.094899 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.094962 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:38.095593 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.096084 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.096818 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.097013 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.097075 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.098281 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:38.098284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:38.098286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:38.098307 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:38.099553 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.099863 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.099865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.099867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.099930 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.100104 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.100107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.100108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.100323 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:38.100416 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.100602 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.100682 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:38.105137 10756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:38.108090 10756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:38.121223 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.116378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:34383 (local address 127.0.0.1:59620)
0927 15:16:38.116867 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.116879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:38.116956 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:38.117591 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:38.117595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:38.117613 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:38.117737 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.117743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.118834 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.118837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.119885 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.119894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.120909 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.120912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.120990 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.121010 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:16:38.121066 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 15:16:38.121115 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":38,"thread_start_us":83,"threads_started":1}
I20260927 15:16:38.121335 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.115797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.9.242.131:44341)
0927 15:16:38.116229 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.116241 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.116264 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:16:38.116452 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.116457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.116662 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.116811 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:38.117290 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.118316 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.119880 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.120588 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.120686 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.120838 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.121091 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.121190 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":148,"threads_started":1}
I20260927 15:16:38.121701 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.115732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:42163 (local address 127.0.0.1:42820)
0927 15:16:38.116400 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.116413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:38.116481 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:38.117787 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:38.117790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:38.117806 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:38.117911 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.117917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.118997 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.119001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.120141 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.120149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.121276 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.121279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.121386 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.121410 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:16:38.121504 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:16:38.121572 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260927 15:16:38.121224 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.115642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:44341 (local address 127.0.0.1:58638)
0927 15:16:38.116083 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.116135 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:38.116256 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:38.116971 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:38.116977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:38.116999 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:38.117123 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.117130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.118439 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.118443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.119741 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.119760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.120703 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.120706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.120808 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.120844 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:16:38.120942 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:16:38.121066 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260927 15:16:38.121807 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.116643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.9.242.129:34383)
0927 15:16:38.117219 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.117231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.117248 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:16:38.117308 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.117313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.117455 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.117598 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:38.117846 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.118703 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.120011 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.120744 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.120858 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.120993 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.121540 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.121636 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":111,"threads_started":1}
I20260927 15:16:38.122116 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.116568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.9.242.130:42163)
0927 15:16:38.117163 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.117176 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.117196 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:16:38.117278 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.117283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.117455 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.117598 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:38.118026 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.118870 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.120255 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.121011 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.121103 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.121218 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.121891 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.121987 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":105,"threads_started":1}
I20260927 15:16:38.124532 11149 tablet_service.cc:1467] Processing CreateTablet for tablet b1fe0b28e1444fed85effdcccaa2e24d (DEFAULT_TABLE table=test-table [id=3873e5eb033844ee8ef3a21e960ab2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:38.124565 11015 tablet_service.cc:1467] Processing CreateTablet for tablet b1fe0b28e1444fed85effdcccaa2e24d (DEFAULT_TABLE table=test-table [id=3873e5eb033844ee8ef3a21e960ab2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:38.124565 10881 tablet_service.cc:1467] Processing CreateTablet for tablet b1fe0b28e1444fed85effdcccaa2e24d (DEFAULT_TABLE table=test-table [id=3873e5eb033844ee8ef3a21e960ab2ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:38.125506 11149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fe0b28e1444fed85effdcccaa2e24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:38.125523 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fe0b28e1444fed85effdcccaa2e24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:38.125522 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fe0b28e1444fed85effdcccaa2e24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:38.132179 11233 tablet_bootstrap.cc:492] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483: Bootstrap starting.
I20260927 15:16:38.132444 11232 tablet_bootstrap.cc:492] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc: Bootstrap starting.
I20260927 15:16:38.133311 11234 tablet_bootstrap.cc:492] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f: Bootstrap starting.
I20260927 15:16:38.134950 11233 tablet_bootstrap.cc:654] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:38.134950 11232 tablet_bootstrap.cc:654] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:38.135298 11234 tablet_bootstrap.cc:654] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:38.135990 11234 log.cc:824] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:38.136106 11232 log.cc:824] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:38.136106 11233 log.cc:824] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:38.137960 11232 tablet_bootstrap.cc:492] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc: No bootstrap required, opened a new log
I20260927 15:16:38.137961 11234 tablet_bootstrap.cc:492] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f: No bootstrap required, opened a new log
I20260927 15:16:38.137961 11233 tablet_bootstrap.cc:492] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483: No bootstrap required, opened a new log
I20260927 15:16:38.138139 11232 ts_tablet_manager.cc:1397] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 15:16:38.138140 11233 ts_tablet_manager.cc:1397] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:16:38.138140 11234 ts_tablet_manager.cc:1397] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:38.141110 11232 raft_consensus.cc:348] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.141360 11232 raft_consensus.cc:374] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:38.141409 11232 raft_consensus.cc:729] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad1e85abdc844238ea28e98ac29bbbc, State: Initialized, Role: FOLLOWER
I20260927 15:16:38.141849 11232 consensus_queue.cc:260] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.142046 11233 raft_consensus.cc:348] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.142311 11233 raft_consensus.cc:374] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:38.142367 11233 raft_consensus.cc:729] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db269fa28c664a2faafff61744b3a483, State: Initialized, Role: FOLLOWER
I20260927 15:16:38.142362 11234 raft_consensus.cc:348] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.142602 11234 raft_consensus.cc:374] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:38.142620 11215 heartbeater.cc:503] Master 127.9.242.190:34381 was elected leader, sending a full tablet report...
I20260927 15:16:38.142659 11234 raft_consensus.cc:729] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed9a297025a40d9a83a4fadd31af16f, State: Initialized, Role: FOLLOWER
I20260927 15:16:38.142858 11232 ts_tablet_manager.cc:1428] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260927 15:16:38.143076 11234 consensus_queue.cc:260] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.143838 11081 heartbeater.cc:503] Master 127.9.242.190:34381 was elected leader, sending a full tablet report...
I20260927 15:16:38.144037 11234 ts_tablet_manager.cc:1428] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260927 15:16:38.144224 11233 consensus_queue.cc:260] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.144819 10947 heartbeater.cc:503] Master 127.9.242.190:34381 was elected leader, sending a full tablet report...
I20260927 15:16:38.145452 11233 ts_tablet_manager.cc:1428] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 15:16:38.151980 10948 tablet.cc:2369] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:38.194087 11240 raft_consensus.cc:482] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:38.194274 11240 raft_consensus.cc:504] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.195355 11240 leader_election.cc:288] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad1e85abdc844238ea28e98ac29bbbc (127.9.242.131:44341), 5ed9a297025a40d9a83a4fadd31af16f (127.9.242.130:42163)
I20260927 15:16:38.200073 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.195676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:43413 (local address 127.9.242.131:44341)
0927 15:16:38.195821 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.195826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.195935 (+ 109us) server_negotiation.cc:407] Connection header received
0927 15:16:38.196059 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.196062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.196113 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.196191 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:38.196444 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.197228 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.198867 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.199475 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.199585 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.199635 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.199882 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.199948 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:16:38.200073 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.195761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:51541 (local address 127.9.242.130:42163)
0927 15:16:38.195880 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.195883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.196056 (+ 173us) server_negotiation.cc:407] Connection header received
0927 15:16:38.196158 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.196162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.196210 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.196277 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:38.196479 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.197216 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.198864 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.199475 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.199585 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.199635 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.199882 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.199948 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:16:38.200122 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.195530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:44341 (local address 127.9.242.129:43413)
0927 15:16:38.195842 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.195855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:38.195962 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:38.196204 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:38.196206 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:38.196219 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:38.196342 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.196349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.197322 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.197326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.198748 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.198760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.199583 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.199587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.199734 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.199802 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 15:16:38.199860 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:16:38.199950 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":139,"thread_start_us":82,"threads_started":1}
I20260927 15:16:38.200608 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.195577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:42163 (local address 127.9.242.129:51541)
0927 15:16:38.195980 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:38.195992 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:38.196077 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:38.196291 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:38.196293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:38.196307 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:38.196392 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.196398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.197344 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.197347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.198766 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:38.198772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.199583 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.199587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:38.199730 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.199786 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 15:16:38.199860 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:16:38.199951 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":153,"thread_start_us":119,"threads_started":1}
I20260927 15:16:38.200727 11169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe0b28e1444fed85effdcccaa2e24d" candidate_uuid: "db269fa28c664a2faafff61744b3a483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad1e85abdc844238ea28e98ac29bbbc" is_pre_election: true
I20260927 15:16:38.201051 11169 raft_consensus.cc:2454] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db269fa28c664a2faafff61744b3a483 in term 0.
I20260927 15:16:38.201193 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe0b28e1444fed85effdcccaa2e24d" candidate_uuid: "db269fa28c664a2faafff61744b3a483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed9a297025a40d9a83a4fadd31af16f" is_pre_election: true
I20260927 15:16:38.201493 11035 raft_consensus.cc:2454] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db269fa28c664a2faafff61744b3a483 in term 0.
I20260927 15:16:38.201522 10836 leader_election.cc:302] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aad1e85abdc844238ea28e98ac29bbbc, db269fa28c664a2faafff61744b3a483; no voters:
I20260927 15:16:38.201812 11240 raft_consensus.cc:2781] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:38.201903 11240 raft_consensus.cc:482] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:38.201946 11240 raft_consensus.cc:3037] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:38.202893 11240 raft_consensus.cc:504] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.203202 11240 leader_election.cc:288] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [CANDIDATE]: Term 1 election: Requested vote from peers aad1e85abdc844238ea28e98ac29bbbc (127.9.242.131:44341), 5ed9a297025a40d9a83a4fadd31af16f (127.9.242.130:42163)
I20260927 15:16:38.203540 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe0b28e1444fed85effdcccaa2e24d" candidate_uuid: "db269fa28c664a2faafff61744b3a483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed9a297025a40d9a83a4fadd31af16f"
I20260927 15:16:38.203639 11035 raft_consensus.cc:3037] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:38.204582 11035 raft_consensus.cc:2454] T b1fe0b28e1444fed85effdcccaa2e24d P 5ed9a297025a40d9a83a4fadd31af16f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db269fa28c664a2faafff61744b3a483 in term 1.
I20260927 15:16:38.204815 11169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fe0b28e1444fed85effdcccaa2e24d" candidate_uuid: "db269fa28c664a2faafff61744b3a483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad1e85abdc844238ea28e98ac29bbbc"
I20260927 15:16:38.204875 10834 leader_election.cc:302] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed9a297025a40d9a83a4fadd31af16f, db269fa28c664a2faafff61744b3a483; no voters:
I20260927 15:16:38.204938 11169 raft_consensus.cc:3037] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:38.205027 11240 raft_consensus.cc:2781] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:38.205242 11240 raft_consensus.cc:686] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [term 1 LEADER]: Becoming Leader. State: Replica: db269fa28c664a2faafff61744b3a483, State: Running, Role: LEADER
I20260927 15:16:38.205559 11240 consensus_queue.cc:237] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } }
I20260927 15:16:38.206048 11169 raft_consensus.cc:2454] T b1fe0b28e1444fed85effdcccaa2e24d P aad1e85abdc844238ea28e98ac29bbbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db269fa28c664a2faafff61744b3a483 in term 1.
I20260927 15:16:38.208730 10756 catalog_manager.cc:5495] T b1fe0b28e1444fed85effdcccaa2e24d P db269fa28c664a2faafff61744b3a483 reported cstate change: term changed from 0 to 1, leader changed from <none> to db269fa28c664a2faafff61744b3a483 (127.9.242.129). New cstate: current_term: 1 leader_uuid: "db269fa28c664a2faafff61744b3a483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad1e85abdc844238ea28e98ac29bbbc" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 44341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed9a297025a40d9a83a4fadd31af16f" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db269fa28c664a2faafff61744b3a483" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34383 } health_report { overall_health: HEALTHY } } }
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:16:38.221794 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.213469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.9.242.129:34383)
0927 15:16:38.213588 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.213593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.213779 (+ 186us) server_negotiation.cc:407] Connection header received
0927 15:16:38.213903 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.213906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.213945 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.214009 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:38.214660 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.215152 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.215979 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.216185 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.216268 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.218640 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:38.218652 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:38.218666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:38.218694 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:38.220100 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.220524 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.220529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.220530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.220608 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.220848 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.220851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.220853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.221103 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:38.221313 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.221488 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.221600 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:16:38.247046 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.239578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59636 (local address 127.9.242.129:34383)
0927 15:16:38.239713 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.239718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.239843 (+ 125us) server_negotiation.cc:407] Connection header received
0927 15:16:38.239958 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.239961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.240017 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.240085 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:38.240741 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.241282 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.242079 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.242297 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.242368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.244574 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:38.244577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:38.244579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:38.244604 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:38.245718 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.246105 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.246108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.246109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.246185 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.246382 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.246384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.246385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.246535 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:38.246643 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.246837 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.246925 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[10711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[10711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:16:38.270573 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:38.262967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.9.242.129:34383)
0927 15:16:38.263103 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:38.263108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:38.263235 (+ 127us) server_negotiation.cc:407] Connection header received
0927 15:16:38.263379 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:38.263382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:38.263425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:38.263493 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:38.264168 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.264665 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.265462 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:38.265725 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:38.265816 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:38.268018 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:38.268021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:38.268023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:38.268048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:38.269098 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.269459 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.269463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.269465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.269535 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:38.269807 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:38.269811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:38.269812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:38.269971 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:38.270102 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:38.270318 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 15:16:38.270425 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 15:16:38.271888 10861 tablet_service.cc:2908] Rejecting scan request for tablet b1fe0b28e1444fed85effdcccaa2e24d: Uninitialized: clean time has not yet been initialized
I20260927 15:16:38.273677 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 10818
I20260927 15:16:38.278523 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 10952
I20260927 15:16:38.282896 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 11086
I20260927 15:16:38.287305 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 10724
2026-09-27T15:16:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2129 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[11257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[11257](info): set up 2 sockets
Sep 27 15:16:38 dist-test-slave-m4lt krb5kdc[11257](info): commencing operation
krb5kdc: starting...
W20260927 15:16:39.241168 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:39 dist-test-slave-m4lt krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:39.354288 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:35815
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:39.459744 11270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:39.460004 11270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:39.460058 11270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:39.464069 11270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:39.464131 11270 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:16:39.464154 11270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:39.464177 11270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:39.464205 11270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:39.464236 11270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:39.469499 11270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:35815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:39.470535 11270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:39.471529 11270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:39.479518 11278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:39.479511 11276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:39.479511 11275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:39.480012 11270 server_base.cc:1034] running on GCE node
I20260927 15:16:39.480513 11270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:39.481603 11270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:39.482790 11270 hybrid_clock.cc:648] HybridClock initialized: now 1790522199482770 us; error 29 us; skew 500 ppm
Sep 27 15:16:39 dist-test-slave-m4lt krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:39.485376 11270 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:39.486294 11270 webserver.cc:460] Webserver started at http://127.9.242.190:44519/ using document root <none> and password file <none>
I20260927 15:16:39.486794 11270 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:39.486850 11270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:39.487015 11270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:39.488937 11270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "ab681f1418fd411ba45d1bf801b08c33"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "1b9236c53a769baa3a85a765057ea013"
server_key_iv: "1695a78517b19b004ef79f46a0349054"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.489429 11270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "ab681f1418fd411ba45d1bf801b08c33"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "1b9236c53a769baa3a85a765057ea013"
server_key_iv: "1695a78517b19b004ef79f46a0349054"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.492882 11270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:16:39.495121 11285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:39.496315 11270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:39.496431 11270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "ab681f1418fd411ba45d1bf801b08c33"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "1b9236c53a769baa3a85a765057ea013"
server_key_iv: "1695a78517b19b004ef79f46a0349054"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.496521 11270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:39.510272 11270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:39.513121 11270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:39.513310 11270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:39.520700 11270 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:35815
I20260927 15:16:39.520705 11347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:35815 every 8 connection(s)
I20260927 15:16:39.521632 11270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:39.524224 11348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:39.529464 11348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: Bootstrap starting.
I20260927 15:16:39.529814 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 11270
I20260927 15:16:39.529923 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:39.530165 10186 external_mini_cluster.cc:1428] Setting key 31b81cef105cb18010af8d4f2f548a39
I20260927 15:16:39.532114 11348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:39.532814 11348 log.cc:824] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:39.534956 11348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: No bootstrap required, opened a new log
Sep 27 15:16:39 dist-test-slave-m4lt krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:39.539157 11348 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681f1418fd411ba45d1bf801b08c33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35815 } }
I20260927 15:16:39.539454 11348 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:39.539513 11348 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab681f1418fd411ba45d1bf801b08c33, State: Initialized, Role: FOLLOWER
I20260927 15:16:39.539956 11348 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681f1418fd411ba45d1bf801b08c33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35815 } }
I20260927 15:16:39.540083 11348 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:39.540143 11348 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:39.540234 11348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:39.541296 11348 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681f1418fd411ba45d1bf801b08c33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35815 } }
I20260927 15:16:39.541671 11348 leader_election.cc:302] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab681f1418fd411ba45d1bf801b08c33; no voters:
I20260927 15:16:39.542019 11348 leader_election.cc:288] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:39.542083 11353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:39.542277 11353 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [term 1 LEADER]: Becoming Leader. State: Replica: ab681f1418fd411ba45d1bf801b08c33, State: Running, Role: LEADER
I20260927 15:16:39.542548 11353 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681f1418fd411ba45d1bf801b08c33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35815 } }
I20260927 15:16:39.543295 11348 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:39.544672 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.531436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.9.242.190:35815)
0927 15:16:39.531834 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:39.531849 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:39.531886 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:16:39.532566 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:39.532585 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:39.532950 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:39.533434 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:39.534259 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.535309 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.536469 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.536939 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.537056 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:39.539731 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:39.539737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:39.539755 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:39.539787 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:39.541681 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:39.542131 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:39.542136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:39.542141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:39.542257 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:39.542650 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:39.542653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:39.542655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:39.543056 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:39.543187 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:39.543592 (+ 405us) server_negotiation.cc:299] Negotiation successful
0927 15:16:39.543878 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260927 15:16:39.546262 11355 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab681f1418fd411ba45d1bf801b08c33. Latest consensus state: current_term: 1 leader_uuid: "ab681f1418fd411ba45d1bf801b08c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681f1418fd411ba45d1bf801b08c33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35815 } } }
I20260927 15:16:39.546253 11354 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab681f1418fd411ba45d1bf801b08c33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab681f1418fd411ba45d1bf801b08c33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35815 } } }
I20260927 15:16:39.546388 11355 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:39.546419 11354 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33 [sys.catalog]: This master's current role is: LEADER
W20260927 15:16:39.548488 11367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:16:39.548550 11367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:16:39.548622 11368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:39.549281 11368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:39.554746 11368 catalog_manager.cc:1347] Generated new cluster ID: 9d2f833776d34f5cb79f4881a0cebcbb
I20260927 15:16:39.554816 11368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:39.568089 11368 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:39.568998 11368 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:39.576870 11368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: Generated new TSK 0
I20260927 15:16:39.577589 11368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:39.608143 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35815
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:39.715785 11374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:39.716028 11374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:39.716077 11374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:39.720350 11374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:39.720425 11374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:39.720465 11374 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:16:39.720496 11374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:39.725793 11374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:35815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:39.726773 11374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:39.727720 11374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:39.737635 11379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:39.737632 11382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:39.737782 11374 server_base.cc:1034] running on GCE node
W20260927 15:16:39.737630 11380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:39.738307 11374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:39.738947 11374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:39.740098 11374 hybrid_clock.cc:648] HybridClock initialized: now 1790522199740078 us; error 35 us; skew 500 ppm
Sep 27 15:16:39 dist-test-slave-m4lt krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:39.742719 11374 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:39.743604 11374 webserver.cc:460] Webserver started at http://127.9.242.129:33215/ using document root <none> and password file <none>
I20260927 15:16:39.744138 11374 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:39.744194 11374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:39.744387 11374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:39.746479 11374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "d48e15a52d054e0792e4e71ab61a1c3f"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "1408e59515f1ec05394b1858f39aa98d"
server_key_iv: "18e570f9c1c5489eaa3aadfdfd5f5b23"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.746968 11374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "d48e15a52d054e0792e4e71ab61a1c3f"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "1408e59515f1ec05394b1858f39aa98d"
server_key_iv: "18e570f9c1c5489eaa3aadfdfd5f5b23"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.750514 11374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:39.752810 11389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:39.754027 11374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:39.754120 11374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "d48e15a52d054e0792e4e71ab61a1c3f"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "1408e59515f1ec05394b1858f39aa98d"
server_key_iv: "18e570f9c1c5489eaa3aadfdfd5f5b23"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.754221 11374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:39.787930 11374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:39.791177 11374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:39.791354 11374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:39.792773 11374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:39.792825 11374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:39.792876 11374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:39.792904 11374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:16:39 dist-test-slave-m4lt krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:39.802942 11374 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:39529
I20260927 15:16:39.804008 11374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:39.804733 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 11374
I20260927 15:16:39.804855 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:39.804836 11504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:39529 every 8 connection(s)
I20260927 15:16:39.805137 10186 external_mini_cluster.cc:1428] Setting key 3e22cfbf3fdbc62f13613272d9b083a7
I20260927 15:16:39.807000 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.794270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35815 (local address 127.9.242.129:48513)
0927 15:16:39.794728 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:39.794783 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:39.795644 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:39.796312 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:39.796318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:39.796626 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:39.797177 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:39.797196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.798082 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.798085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:39.798944 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:39.798952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.799507 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.799512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:39.799725 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:39.800394 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:39.800412 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:39.802269 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:39.804174 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:39.804180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:39.804195 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:39.804432 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:39.804739 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:39.804743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:39.804745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:39.804906 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:39.805350 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:39.805355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:39.805632 (+ 277us) client_negotiation.cc:769] Sending connection context
0927 15:16:39.805837 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 15:16:39.806153 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260927 15:16:39.807591 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.793696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:48513 (local address 127.9.242.190:35815)
0927 15:16:39.793824 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:39.793827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:39.794886 (+ 1059us) server_negotiation.cc:407] Connection header received
0927 15:16:39.795896 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:39.795900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:39.795955 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:39.796000 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:39.797316 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.797967 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.799052 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.799329 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.799431 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:39.802425 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:39.802428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:39.802431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:39.802457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:39.804055 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:39.804535 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:39.804539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:39.804540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:39.804632 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:39.805020 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:39.805024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:39.805026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:39.805262 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:39.805355 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:39.807369 (+ 2014us) server_negotiation.cc:299] Negotiation successful
0927 15:16:39.807472 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:16:39.808722 11505 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35815
I20260927 15:16:39.808856 11505 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:39.809372 11505 heartbeater.cc:511] Master 127.9.242.190:35815 requested a full tablet report, sending...
I20260927 15:16:39.810940 11301 ts_manager.cc:194] Registered new tserver with Master: d48e15a52d054e0792e4e71ab61a1c3f (127.9.242.129:39529)
I20260927 15:16:39.812049 11301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:48513
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:16:39.820200 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.810813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:36417 (local address 127.9.242.190:35815)
0927 15:16:39.810924 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:39.810929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:39.813980 (+ 3051us) server_negotiation.cc:407] Connection header received
0927 15:16:39.814079 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:39.814083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:39.814123 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:39.814184 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:39.814604 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.815167 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.816056 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.816307 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.816386 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:39.817384 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:39.817388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:39.817391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:39.817417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:39.818640 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:39.819050 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:39.819052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:39.819053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:39.819177 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:39.819427 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:39.819430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:39.819431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:39.819615 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:39.819689 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:39.819992 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 15:16:39.820088 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:16:39.820256 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:39.810749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35815 (local address 127.9.242.129:36417)
0927 15:16:39.813895 (+ 3146us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:39.813908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:39.814000 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:39.814210 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:39.814213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:39.814375 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:39.814497 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:39.814504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.815278 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.815281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:39.815949 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:39.815959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:39.816412 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:39.816415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:39.816498 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:39.816937 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:39.816948 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:39.817237 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:39.818746 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:39.818749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:39.818751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:39.818974 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:39.819259 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:39.819263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:39.819264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:39.819349 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:39.819699 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:39.819702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:39.819800 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 15:16:39.819908 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:16:39.820042 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3071,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:39.832803 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35815
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:39.939797 11514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:39.940037 11514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:39.940078 11514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:39.944137 11514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:39.944195 11514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:39.944232 11514 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:16:39.944270 11514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:39.949492 11514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:35815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:39.950477 11514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:39.951392 11514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:39.960619 11522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:39.960613 11519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:39.960667 11514 server_base.cc:1034] running on GCE node
W20260927 15:16:39.960616 11520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:39.961206 11514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:39.961903 11514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:39.963055 11514 hybrid_clock.cc:648] HybridClock initialized: now 1790522199963046 us; error 29 us; skew 500 ppm
Sep 27 15:16:39 dist-test-slave-m4lt krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:39.965624 11514 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:39.966464 11514 webserver.cc:460] Webserver started at http://127.9.242.130:36179/ using document root <none> and password file <none>
I20260927 15:16:39.966984 11514 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:39.967044 11514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:39.967228 11514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:39.969561 11514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "889c030fe6e1425db1923b7489f94159"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "651c6a736d2760e4bf250d33285f5332"
server_key_iv: "4f6cf96f19a750c0844da68cc7147ffe"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.970230 11514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "889c030fe6e1425db1923b7489f94159"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "651c6a736d2760e4bf250d33285f5332"
server_key_iv: "4f6cf96f19a750c0844da68cc7147ffe"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.974112 11514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:16:39.976387 11529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:39.977591 11514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:39.977735 11514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "889c030fe6e1425db1923b7489f94159"
format_stamp: "Formatted at 2026-09-27 15:16:39 on dist-test-slave-m4lt"
server_key: "651c6a736d2760e4bf250d33285f5332"
server_key_iv: "4f6cf96f19a750c0844da68cc7147ffe"
server_key_version: "encryptionkey@0"
I20260927 15:16:39.977841 11514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:39.995622 11514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:39.998709 11514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:39.998875 11514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:40.000357 11514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:40.000407 11514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:40.000447 11514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:40.000474 11514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:40.010437 11514 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:44863
I20260927 15:16:40.010489 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:44863 every 8 connection(s)
I20260927 15:16:40.011330 11514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:40.014256 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.001218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:38323 (local address 127.9.242.190:35815)
0927 15:16:40.001354 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.001360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.002291 (+ 931us) server_negotiation.cc:407] Connection header received
0927 15:16:40.003375 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.003379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.003435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.003516 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.005097 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.005941 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.007342 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.007661 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.007778 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.010629 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.010633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.010636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.010679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.012196 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.012684 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.012687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.012689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.012774 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.013060 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.013062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.013063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.013317 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.013397 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.013932 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.014134 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:16:40.015014 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.001631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35815 (local address 127.9.242.130:38323)
0927 15:16:40.002126 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.002177 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.003131 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.003715 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.003724 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.004204 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:40.004921 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.004940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.006075 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.006079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.007202 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.007212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.007826 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.007829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.007978 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.008678 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:40.008699 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:40.010400 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:40.012313 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.012318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.012330 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.012571 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.012857 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.012859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.012860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.012985 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.013398 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:40.013403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:40.013628 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.013845 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.014104 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
I20260927 15:16:40.016330 11645 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35815
I20260927 15:16:40.016479 11645 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:40.016956 11645 heartbeater.cc:511] Master 127.9.242.190:35815 requested a full tablet report, sending...
I20260927 15:16:40.017997 11302 ts_manager.cc:194] Registered new tserver with Master: 889c030fe6e1425db1923b7489f94159 (127.9.242.130:44863)
I20260927 15:16:40.018697 11302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:38323
I20260927 15:16:40.018777 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 11514
I20260927 15:16:40.018874 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:40.019110 10186 external_mini_cluster.cc:1428] Setting key 4f364059470d4ace950f271902757918
I20260927 15:16:40.025326 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.018296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35815 (local address 127.9.242.130:52147)
0927 15:16:40.018577 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.018588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.018679 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.018976 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.018979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.019156 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:40.019268 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.019274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.020124 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.020127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.020824 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.020833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.021255 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.021259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.021344 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.021874 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:40.021889 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:40.022218 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:40.023881 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.023884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.023885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.024092 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.024426 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.024429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.024430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.024500 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.024929 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:40.024931 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:40.025000 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.025071 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.025175 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":45,"thread_start_us":104,"threads_started":1}
I20260927 15:16:40.025420 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.018393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:52147 (local address 127.9.242.190:35815)
0927 15:16:40.018507 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.018511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.018658 (+ 147us) server_negotiation.cc:407] Connection header received
0927 15:16:40.018756 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.018759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.018804 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.018869 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.019370 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.020005 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.020933 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.021159 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.021242 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.022372 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.022375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.022378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.022404 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.023788 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.024247 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.024251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.024253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.024340 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.024576 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.024579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.024580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.024849 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.024948 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.025154 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.025266 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:40.044916 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35815
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:40.153069 11654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:40.153353 11654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:40.153401 11654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:40.157502 11654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:40.157557 11654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:40.157603 11654 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:16:40.157670 11654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:40.162943 11654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:35815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:40.163928 11654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:40.164904 11654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:40.174338 11662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:40.174387 11654 server_base.cc:1034] running on GCE node
W20260927 15:16:40.174340 11660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:40.174338 11659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:40.174973 11654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:40.175604 11654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:40.176764 11654 hybrid_clock.cc:648] HybridClock initialized: now 1790522200176747 us; error 32 us; skew 500 ppm
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522200, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:40.179646 11654 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:40.180585 11654 webserver.cc:460] Webserver started at http://127.9.242.131:40383/ using document root <none> and password file <none>
I20260927 15:16:40.181102 11654 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:40.181174 11654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:40.181355 11654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:40.183429 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "f5a7be2b2859460c88b132c8db009f2d"
format_stamp: "Formatted at 2026-09-27 15:16:40 on dist-test-slave-m4lt"
server_key: "d56e68309d0f7301f40c2a6942f6e96f"
server_key_iv: "c756464d893033fc75fd92cedcee593b"
server_key_version: "encryptionkey@0"
I20260927 15:16:40.183923 11654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "f5a7be2b2859460c88b132c8db009f2d"
format_stamp: "Formatted at 2026-09-27 15:16:40 on dist-test-slave-m4lt"
server_key: "d56e68309d0f7301f40c2a6942f6e96f"
server_key_iv: "c756464d893033fc75fd92cedcee593b"
server_key_version: "encryptionkey@0"
I20260927 15:16:40.187749 11654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:16:40.190505 11669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:40.191767 11654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:40.191869 11654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "f5a7be2b2859460c88b132c8db009f2d"
format_stamp: "Formatted at 2026-09-27 15:16:40 on dist-test-slave-m4lt"
server_key: "d56e68309d0f7301f40c2a6942f6e96f"
server_key_iv: "c756464d893033fc75fd92cedcee593b"
server_key_version: "encryptionkey@0"
I20260927 15:16:40.191958 11654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:40.202828 11654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:40.205883 11654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:40.206029 11654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:40.207427 11654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:40.207476 11654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:40.207515 11654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:40.207543 11654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522200, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:40.217465 11654 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:35845
I20260927 15:16:40.217603 11784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:35845 every 8 connection(s)
I20260927 15:16:40.218442 11654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:40.220118 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.208097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:53103 (local address 127.9.242.190:35815)
0927 15:16:40.208286 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.208293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.209145 (+ 852us) server_negotiation.cc:407] Connection header received
0927 15:16:40.210332 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.210335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.210390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.210465 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.211702 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.212224 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.213407 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.213640 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.213748 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.216547 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.216550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.216552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.216577 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.218115 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.218658 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.218661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.218662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.218735 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.219125 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.219128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.219130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.219356 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.219454 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.219902 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.219999 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:16:40.220479 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 11654
I20260927 15:16:40.220594 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:40.220860 10186 external_mini_cluster.cc:1428] Setting key ff44421ab725592bde26004368dcc345
I20260927 15:16:40.221076 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.208548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35815 (local address 127.9.242.131:53103)
0927 15:16:40.208991 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.209040 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.210117 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.210634 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.210642 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.211037 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:40.211555 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.211569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.212336 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.212339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.213292 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.213301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.213893 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.213896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.214028 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.214597 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:40.214613 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:40.216350 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:40.218227 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.218233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.218248 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.218563 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.218829 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.218833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.218834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.218993 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.219458 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:40.219463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:40.219712 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.219861 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.220108 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260927 15:16:40.222358 11785 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35815
I20260927 15:16:40.222500 11785 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:40.223068 11785 heartbeater.cc:511] Master 127.9.242.190:35815 requested a full tablet report, sending...
I20260927 15:16:40.224063 11301 ts_manager.cc:194] Registered new tserver with Master: f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845)
I20260927 15:16:40.224625 11301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:53103
I20260927 15:16:40.231177 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.225074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:33133 (local address 127.9.242.190:35815)
0927 15:16:40.225177 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.225181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.225334 (+ 153us) server_negotiation.cc:407] Connection header received
0927 15:16:40.225460 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.225463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.225505 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.225563 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.226034 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.226530 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.227353 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.227573 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.227656 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.228602 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.228606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.228609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.228632 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.229773 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.230127 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.230131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.230133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.230208 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.230439 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.230442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.230443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.230609 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.230691 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.230931 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.231027 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:16:40.231196 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.225025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35815 (local address 127.9.242.131:33133)
0927 15:16:40.225261 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.225273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.225371 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.225587 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.225590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.225833 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:40.225931 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.225937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.226645 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.226648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.227246 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.227255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.227667 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.227670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.227757 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.228176 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:40.228186 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:40.228481 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:40.229863 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.229866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.229869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.230047 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.230283 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.230286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.230288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.230364 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.230691 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:40.230694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:40.230788 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.230873 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.231008 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260927 15:16:40.234453 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:40.258055 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.250372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.9.242.190:35815)
0927 15:16:40.250481 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.250485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.250497 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:16:40.250536 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.250539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.250576 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.250644 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.251234 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.251752 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.252568 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.252779 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.252857 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.255324 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.255327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.255329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.255352 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.256589 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.256939 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.256941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.256943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.257006 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.257219 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.257220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.257222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.257387 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.257490 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.257836 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.257936 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:16:40.260726 11301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:40.263736 11301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:40.276932 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.271647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.9.242.130:44863)
0927 15:16:40.272049 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.272060 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.272084 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:16:40.272161 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.272166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.272397 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.272902 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:40.273038 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.273967 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.275512 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.276299 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.276392 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.276546 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.276701 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.276795 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260927 15:16:40.276921 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.271480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44863 (local address 127.0.0.1:42582)
0927 15:16:40.271800 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.271850 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.271970 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.272569 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.272575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.272599 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:40.272743 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.272751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.274067 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.274070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.275387 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.275397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.276412 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.276416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.276515 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.276551 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.276658 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.276773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260927 15:16:40.277199 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.272570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.9.242.131:35845)
0927 15:16:40.272853 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.272864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.273170 (+ 306us) server_negotiation.cc:407] Connection header received
0927 15:16:40.273233 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.273237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.273385 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.273496 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:40.273849 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.274625 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.275812 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.276314 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.276404 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.276557 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.276989 (+ 432us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.277076 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260927 15:16:40.277420 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.272209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:35845 (local address 127.0.0.1:38402)
0927 15:16:40.272993 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.273005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.273090 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.273611 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.273614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.273630 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:40.273743 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.273749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.274744 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.274748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.275703 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.275711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.276618 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.276621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.276701 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.276722 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.276792 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.276859 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260927 15:16:40.277647 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.271781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:39529 (local address 127.0.0.1:54482)
0927 15:16:40.272941 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.272953 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.273038 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.273470 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.273473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.273491 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:40.273571 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.273577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.274798 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.274801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.275943 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.275952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.277312 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.277316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.277398 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.277420 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.277480 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.277540 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260927 15:16:40.278270 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.272268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.9.242.129:39529)
0927 15:16:40.272513 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.272526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.273030 (+ 504us) server_negotiation.cc:407] Connection header received
0927 15:16:40.273138 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.273143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.273339 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.273480 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:40.273679 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.274667 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.276063 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.276768 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.276848 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.278015 (+ 1167us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.278061 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.278139 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260927 15:16:40.280063 11719 tablet_service.cc:1467] Processing CreateTablet for tablet cfbd78e84adb49c8a7219de693125f61 (DEFAULT_TABLE table=test-table [id=2215968dca814bc4b25efd1f3c629339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:40.280064 11579 tablet_service.cc:1467] Processing CreateTablet for tablet cfbd78e84adb49c8a7219de693125f61 (DEFAULT_TABLE table=test-table [id=2215968dca814bc4b25efd1f3c629339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:40.280375 11439 tablet_service.cc:1467] Processing CreateTablet for tablet cfbd78e84adb49c8a7219de693125f61 (DEFAULT_TABLE table=test-table [id=2215968dca814bc4b25efd1f3c629339]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:40.281158 11719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfbd78e84adb49c8a7219de693125f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.281153 11579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfbd78e84adb49c8a7219de693125f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.281284 11439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfbd78e84adb49c8a7219de693125f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.287935 11808 tablet_bootstrap.cc:492] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Bootstrap starting.
I20260927 15:16:40.289177 11810 tablet_bootstrap.cc:492] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Bootstrap starting.
I20260927 15:16:40.289985 11809 tablet_bootstrap.cc:492] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Bootstrap starting.
I20260927 15:16:40.290012 11808 tablet_bootstrap.cc:654] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.290699 11808 log.cc:824] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:40.291251 11810 tablet_bootstrap.cc:654] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.291996 11809 tablet_bootstrap.cc:654] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.292032 11810 log.cc:824] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:40.292330 11808 tablet_bootstrap.cc:492] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: No bootstrap required, opened a new log
I20260927 15:16:40.292505 11808 ts_tablet_manager.cc:1397] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:16:40.292769 11809 log.cc:824] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:40.293767 11810 tablet_bootstrap.cc:492] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: No bootstrap required, opened a new log
I20260927 15:16:40.293931 11810 ts_tablet_manager.cc:1397] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:40.295451 11808 raft_consensus.cc:348] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.295650 11808 raft_consensus.cc:374] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.295701 11808 raft_consensus.cc:729] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48e15a52d054e0792e4e71ab61a1c3f, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.296077 11808 consensus_queue.cc:260] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.296682 11505 heartbeater.cc:503] Master 127.9.242.190:35815 was elected leader, sending a full tablet report...
I20260927 15:16:40.296943 11808 ts_tablet_manager.cc:1428] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 15:16:40.296940 11810 raft_consensus.cc:348] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.297168 11810 raft_consensus.cc:374] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.297222 11810 raft_consensus.cc:729] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889c030fe6e1425db1923b7489f94159, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.297612 11810 consensus_queue.cc:260] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.298235 11645 heartbeater.cc:503] Master 127.9.242.190:35815 was elected leader, sending a full tablet report...
I20260927 15:16:40.298573 11810 ts_tablet_manager.cc:1428] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:40.298859 11809 tablet_bootstrap.cc:492] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: No bootstrap required, opened a new log
I20260927 15:16:40.298995 11809 ts_tablet_manager.cc:1397] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260927 15:16:40.302366 11809 raft_consensus.cc:348] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.302562 11809 raft_consensus.cc:374] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.302594 11809 raft_consensus.cc:729] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a7be2b2859460c88b132c8db009f2d, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.302894 11809 consensus_queue.cc:260] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.303421 11785 heartbeater.cc:503] Master 127.9.242.190:35815 was elected leader, sending a full tablet report...
I20260927 15:16:40.303803 11809 ts_tablet_manager.cc:1428] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20260927 15:16:40.305130 11506 tablet.cc:2369] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:40.365774 11814 raft_consensus.cc:482] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:40.365999 11814 raft_consensus.cc:504] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.367116 11814 leader_election.cc:288] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889c030fe6e1425db1923b7489f94159 (127.9.242.130:44863), f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845)
I20260927 15:16:40.370865 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.367448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:36171 (local address 127.9.242.130:44863)
0927 15:16:40.367561 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.367565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.367624 (+ 59us) server_negotiation.cc:407] Connection header received
0927 15:16:40.367803 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.367807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.367861 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.367906 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:40.368244 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.368746 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.369833 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.370226 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.370333 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.370386 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.370570 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.370646 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:16:40.370993 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.367780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:32951 (local address 127.9.242.131:35845)
0927 15:16:40.367893 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.367897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.367978 (+ 81us) server_negotiation.cc:407] Connection header received
0927 15:16:40.368120 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.368123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.368165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.368222 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:40.368479 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.369005 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.369976 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.370399 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.370507 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.370554 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.370812 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.370890 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:40.371224 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.367313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44863 (local address 127.9.242.129:36171)
0927 15:16:40.367548 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.367562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.367663 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.368026 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.368029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.368044 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:40.368138 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.368143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.368881 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.368885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.369706 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.369715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.370339 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.370343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.370433 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.370455 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.371011 (+ 556us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.371090 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260927 15:16:40.371307 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.367653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:35845 (local address 127.9.242.129:32951)
0927 15:16:40.367893 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.367906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.368015 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.368237 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.368240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.368282 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:40.368388 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.368393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.369101 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.369105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.369873 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.369881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.370500 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.370503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.370597 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.370618 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.371107 (+ 489us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.371164 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":62,"threads_started":1}
I20260927 15:16:40.371883 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbd78e84adb49c8a7219de693125f61" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a7be2b2859460c88b132c8db009f2d" is_pre_election: true
I20260927 15:16:40.371882 11599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbd78e84adb49c8a7219de693125f61" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889c030fe6e1425db1923b7489f94159" is_pre_election: true
I20260927 15:16:40.372210 11739 raft_consensus.cc:2454] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 0.
I20260927 15:16:40.372210 11599 raft_consensus.cc:2454] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 0.
I20260927 15:16:40.372740 11392 leader_election.cc:302] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d48e15a52d054e0792e4e71ab61a1c3f, f5a7be2b2859460c88b132c8db009f2d; no voters:
I20260927 15:16:40.373023 11814 raft_consensus.cc:2781] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:40.373112 11814 raft_consensus.cc:482] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:40.373160 11814 raft_consensus.cc:3037] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.374208 11814 raft_consensus.cc:504] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.374526 11814 leader_election.cc:288] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 election: Requested vote from peers 889c030fe6e1425db1923b7489f94159 (127.9.242.130:44863), f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845)
I20260927 15:16:40.374902 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbd78e84adb49c8a7219de693125f61" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a7be2b2859460c88b132c8db009f2d"
I20260927 15:16:40.374902 11599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfbd78e84adb49c8a7219de693125f61" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889c030fe6e1425db1923b7489f94159"
I20260927 15:16:40.375031 11739 raft_consensus.cc:3037] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.375031 11599 raft_consensus.cc:3037] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.376622 11739 raft_consensus.cc:2454] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 1.
I20260927 15:16:40.376650 11599 raft_consensus.cc:2454] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 1.
I20260927 15:16:40.376953 11392 leader_election.cc:302] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d48e15a52d054e0792e4e71ab61a1c3f, f5a7be2b2859460c88b132c8db009f2d; no voters:
I20260927 15:16:40.377120 11814 raft_consensus.cc:2781] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:40.377296 11814 raft_consensus.cc:686] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 LEADER]: Becoming Leader. State: Replica: d48e15a52d054e0792e4e71ab61a1c3f, State: Running, Role: LEADER
I20260927 15:16:40.377614 11814 consensus_queue.cc:237] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } }
I20260927 15:16:40.380777 11301 catalog_manager.cc:5495] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d48e15a52d054e0792e4e71ab61a1c3f (127.9.242.129). New cstate: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } health_report { overall_health: HEALTHY } } }
I20260927 15:16:40.403213 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.399524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.9.242.129:39529)
0927 15:16:40.399686 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.399689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.399704 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:16:40.399896 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.399899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.399954 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.400032 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:40.400318 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.400848 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.401814 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.402019 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.402090 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.402415 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.402824 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.402909 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.403038 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.403095 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:16:40.409183 11739 raft_consensus.cc:1261] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Refusing update from remote peer d48e15a52d054e0792e4e71ab61a1c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:40.409343 11599 raft_consensus.cc:1261] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Refusing update from remote peer d48e15a52d054e0792e4e71ab61a1c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:40.409878 11814 consensus_queue.cc:1035] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:40.410070 11819 consensus_queue.cc:1035] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:40.421452 11823 mvcc.cc:205] Tried to move back new op lower bound from 7333978932869177344 to 7333978932749713408. Current Snapshot: MvccSnapshot[applied={T|T < 7333978932869177344 or (T in {7333978932869177344})}]
I20260927 15:16:40.429145 11302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:40.429299 11302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:40.431528 11302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: Sending DeleteTablet for 3 replicas of tablet cfbd78e84adb49c8a7219de693125f61
I20260927 15:16:40.432260 11579 tablet_service.cc:1514] Processing DeleteTablet for tablet cfbd78e84adb49c8a7219de693125f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:40 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:42582
I20260927 15:16:40.432260 11719 tablet_service.cc:1514] Processing DeleteTablet for tablet cfbd78e84adb49c8a7219de693125f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:40 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:38402
I20260927 15:16:40.432327 11439 tablet_service.cc:1514] Processing DeleteTablet for tablet cfbd78e84adb49c8a7219de693125f61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:40 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54482
I20260927 15:16:40.432744 11833 tablet_replica.cc:330] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: stopping tablet replica
I20260927 15:16:40.432744 11832 tablet_replica.cc:330] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: stopping tablet replica
I20260927 15:16:40.433040 11832 raft_consensus.cc:2229] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:40.433040 11833 raft_consensus.cc:2229] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:40.433339 11833 raft_consensus.cc:2258] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:40.433339 11832 raft_consensus.cc:2258] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:40.433854 11834 tablet_replica.cc:330] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: stopping tablet replica
I20260927 15:16:40.434176 11834 raft_consensus.cc:2229] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:40.434084 11302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:40.434494 11302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:40.434741 11834 raft_consensus.cc:2258] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:40.434916 11833 ts_tablet_manager.cc:1905] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:40.436036 11834 ts_tablet_manager.cc:1905] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:40.437537 11833 ts_tablet_manager.cc:1918] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:40.437634 11833 log.cc:1196] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/cfbd78e84adb49c8a7219de693125f61
I20260927 15:16:40.437964 11833 ts_tablet_manager.cc:1939] T cfbd78e84adb49c8a7219de693125f61 P f5a7be2b2859460c88b132c8db009f2d: Deleting consensus metadata
I20260927 15:16:40.438793 11834 ts_tablet_manager.cc:1918] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:40.438936 11834 log.cc:1196] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/cfbd78e84adb49c8a7219de693125f61
I20260927 15:16:40.439123 11288 catalog_manager.cc:4841] TS f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845): tablet cfbd78e84adb49c8a7219de693125f61 (table test-table [id=2215968dca814bc4b25efd1f3c629339]) successfully deleted
I20260927 15:16:40.439311 11834 ts_tablet_manager.cc:1939] T cfbd78e84adb49c8a7219de693125f61 P d48e15a52d054e0792e4e71ab61a1c3f: Deleting consensus metadata
I20260927 15:16:40.439747 11719 tablet_service.cc:1467] Processing CreateTablet for tablet 46ddd825716442cf98fbea696e08a787 (DEFAULT_TABLE table=test-table [id=c08a4d48063a4c26a56dfca1f81eda40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:40.439749 11579 tablet_service.cc:1467] Processing CreateTablet for tablet 46ddd825716442cf98fbea696e08a787 (DEFAULT_TABLE table=test-table [id=c08a4d48063a4c26a56dfca1f81eda40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:40.439983 11579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ddd825716442cf98fbea696e08a787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.439983 11719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ddd825716442cf98fbea696e08a787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.442034 11809 tablet_bootstrap.cc:492] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Bootstrap starting.
I20260927 15:16:40.442205 11810 tablet_bootstrap.cc:492] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Bootstrap starting.
I20260927 15:16:40.442912 11809 tablet_bootstrap.cc:654] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.442927 11810 tablet_bootstrap.cc:654] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.443603 11289 catalog_manager.cc:4841] TS d48e15a52d054e0792e4e71ab61a1c3f (127.9.242.129:39529): tablet cfbd78e84adb49c8a7219de693125f61 (table test-table [id=2215968dca814bc4b25efd1f3c629339]) successfully deleted
I20260927 15:16:40.443912 11439 tablet_service.cc:1467] Processing CreateTablet for tablet 46ddd825716442cf98fbea696e08a787 (DEFAULT_TABLE table=test-table [id=c08a4d48063a4c26a56dfca1f81eda40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:40.444180 11439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ddd825716442cf98fbea696e08a787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.444362 11810 tablet_bootstrap.cc:492] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: No bootstrap required, opened a new log
I20260927 15:16:40.444459 11810 ts_tablet_manager.cc:1397] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:40.444865 11810 raft_consensus.cc:348] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.444957 11810 raft_consensus.cc:374] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.445006 11810 raft_consensus.cc:729] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889c030fe6e1425db1923b7489f94159, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.445132 11810 consensus_queue.cc:260] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.445365 11810 ts_tablet_manager.cc:1428] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:40.446317 11808 tablet_bootstrap.cc:492] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: Bootstrap starting.
I20260927 15:16:40.446528 11832 ts_tablet_manager.cc:1905] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:40.447211 11808 tablet_bootstrap.cc:654] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.449374 11832 ts_tablet_manager.cc:1918] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:40.449465 11832 log.cc:1196] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/cfbd78e84adb49c8a7219de693125f61
I20260927 15:16:40.449847 11832 ts_tablet_manager.cc:1939] T cfbd78e84adb49c8a7219de693125f61 P 889c030fe6e1425db1923b7489f94159: Deleting consensus metadata
I20260927 15:16:40.449890 11809 tablet_bootstrap.cc:492] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: No bootstrap required, opened a new log
I20260927 15:16:40.449973 11809 ts_tablet_manager.cc:1397] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260927 15:16:40.450462 11809 raft_consensus.cc:348] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.450577 11809 raft_consensus.cc:374] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.450613 11809 raft_consensus.cc:729] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a7be2b2859460c88b132c8db009f2d, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.450734 11809 consensus_queue.cc:260] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.450987 11809 ts_tablet_manager.cc:1428] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:40.451467 11288 catalog_manager.cc:4841] TS 889c030fe6e1425db1923b7489f94159 (127.9.242.130:44863): tablet cfbd78e84adb49c8a7219de693125f61 (table test-table [id=2215968dca814bc4b25efd1f3c629339]) successfully deleted
I20260927 15:16:40.452214 11808 tablet_bootstrap.cc:492] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: No bootstrap required, opened a new log
I20260927 15:16:40.452301 11808 ts_tablet_manager.cc:1397] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:16:40.453255 11808 raft_consensus.cc:348] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.453359 11808 raft_consensus.cc:374] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.453413 11808 raft_consensus.cc:729] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48e15a52d054e0792e4e71ab61a1c3f, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.453555 11808 consensus_queue.cc:260] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.454200 11808 ts_tablet_manager.cc:1428] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260927 15:16:40.469368 11786 tablet.cc:2369] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:40.473433 11815 raft_consensus.cc:482] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:40.473553 11815 raft_consensus.cc:504] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.474574 11815 leader_election.cc:288] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d48e15a52d054e0792e4e71ab61a1c3f (127.9.242.129:39529), f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845)
I20260927 15:16:40.478926 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.475171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:45255 (local address 127.9.242.129:39529)
0927 15:16:40.475305 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.475310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.475323 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:16:40.475374 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.475376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.475422 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.475485 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:40.475844 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.476449 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.477808 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.478411 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.478525 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.478578 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.478746 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.478813 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:16:40.478926 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.474804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:39529 (local address 127.9.242.130:45255)
0927 15:16:40.474996 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.475011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.475097 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.475589 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.475593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.475608 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:40.475715 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.475721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.476552 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.476555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.477688 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.477699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.478524 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.478528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.478626 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.478648 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.478713 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.478798 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 15:16:40.479104 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.474954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:57253 (local address 127.9.242.131:35845)
0927 15:16:40.475057 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.475061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.475358 (+ 297us) server_negotiation.cc:407] Connection header received
0927 15:16:40.475522 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.475525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.475568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.475610 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:40.476056 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.476776 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.478086 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.478472 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.478576 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.478623 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.478910 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.478985 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:16:40.479436 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.474808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:35845 (local address 127.9.242.130:57253)
0927 15:16:40.475270 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.475291 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.475416 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.475720 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.475722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.475736 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:40.475941 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.475950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.476888 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.476891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.477986 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.477993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.478663 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.478667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.478773 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.478795 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.479215 (+ 420us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.479299 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":201,"thread_start_us":92,"threads_started":1}
I20260927 15:16:40.479729 11459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ddd825716442cf98fbea696e08a787" candidate_uuid: "889c030fe6e1425db1923b7489f94159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" is_pre_election: true
I20260927 15:16:40.479787 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ddd825716442cf98fbea696e08a787" candidate_uuid: "889c030fe6e1425db1923b7489f94159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a7be2b2859460c88b132c8db009f2d" is_pre_election: true
I20260927 15:16:40.479913 11739 raft_consensus.cc:2454] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889c030fe6e1425db1923b7489f94159 in term 0.
I20260927 15:16:40.479976 11459 raft_consensus.cc:2454] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889c030fe6e1425db1923b7489f94159 in term 0.
I20260927 15:16:40.480291 11532 leader_election.cc:302] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889c030fe6e1425db1923b7489f94159, f5a7be2b2859460c88b132c8db009f2d; no voters:
I20260927 15:16:40.480527 11815 raft_consensus.cc:2781] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:40.480585 11815 raft_consensus.cc:482] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:40.480624 11815 raft_consensus.cc:3037] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.481637 11815 raft_consensus.cc:504] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.481998 11815 leader_election.cc:288] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [CANDIDATE]: Term 1 election: Requested vote from peers d48e15a52d054e0792e4e71ab61a1c3f (127.9.242.129:39529), f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845)
I20260927 15:16:40.482326 11459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ddd825716442cf98fbea696e08a787" candidate_uuid: "889c030fe6e1425db1923b7489f94159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48e15a52d054e0792e4e71ab61a1c3f"
I20260927 15:16:40.482362 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ddd825716442cf98fbea696e08a787" candidate_uuid: "889c030fe6e1425db1923b7489f94159" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a7be2b2859460c88b132c8db009f2d"
I20260927 15:16:40.482440 11459 raft_consensus.cc:3037] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.482512 11739 raft_consensus.cc:3037] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.483350 11459 raft_consensus.cc:2454] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889c030fe6e1425db1923b7489f94159 in term 1.
I20260927 15:16:40.483541 11739 raft_consensus.cc:2454] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889c030fe6e1425db1923b7489f94159 in term 1.
I20260927 15:16:40.483623 11533 leader_election.cc:302] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889c030fe6e1425db1923b7489f94159, d48e15a52d054e0792e4e71ab61a1c3f; no voters:
I20260927 15:16:40.483780 11815 raft_consensus.cc:2781] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:40.483976 11815 raft_consensus.cc:686] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 1 LEADER]: Becoming Leader. State: Replica: 889c030fe6e1425db1923b7489f94159, State: Running, Role: LEADER
I20260927 15:16:40.484283 11815 consensus_queue.cc:237] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.486685 11302 catalog_manager.cc:5495] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889c030fe6e1425db1923b7489f94159 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "889c030fe6e1425db1923b7489f94159" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } health_report { overall_health: UNKNOWN } } }
W20260927 15:16:40.512427 11646 tablet.cc:2369] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522199, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:40.523339 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.515215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35815 (local address 127.0.0.1:47294)
0927 15:16:40.515479 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.515494 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.515595 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.515870 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.515873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.516291 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:40.516381 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.516387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.517377 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.517379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.517976 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.517984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.518512 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.518515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.518577 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.519093 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:40.519104 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:40.520366 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:40.521883 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.521892 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.521897 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.522096 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.522331 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:40.522334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:40.522336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:40.522415 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:40.522813 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:40.522819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:40.522923 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.523014 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.523156 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
I20260927 15:16:40.523339 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.515307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.9.242.190:35815)
0927 15:16:40.515437 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.515442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.515562 (+ 120us) server_negotiation.cc:407] Connection header received
0927 15:16:40.515673 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.515676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.515724 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.515818 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.516489 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.517280 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.518081 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.518409 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.518510 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.520484 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.520487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.520489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.520512 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.521796 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.522179 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.522182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.522184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.522251 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.522479 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.522482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.522484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.522683 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.522763 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.523072 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.523209 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":46}
I20260927 15:16:40.526974 11302 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:47294:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:16:40.527671 11302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:40.532300 11439 tablet_service.cc:1467] Processing CreateTablet for tablet 77310e5469ee4ec79a1736a417ea67a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=166517b57d9143e2bba843f738aca7a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:16:40.532436 11579 tablet_service.cc:1467] Processing CreateTablet for tablet 77310e5469ee4ec79a1736a417ea67a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=166517b57d9143e2bba843f738aca7a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:16:40.532459 11719 tablet_service.cc:1467] Processing CreateTablet for tablet 77310e5469ee4ec79a1736a417ea67a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=166517b57d9143e2bba843f738aca7a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:16:40.532677 11439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77310e5469ee4ec79a1736a417ea67a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.532783 11719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77310e5469ee4ec79a1736a417ea67a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.532791 11579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77310e5469ee4ec79a1736a417ea67a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:40.535440 11808 tablet_bootstrap.cc:492] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: Bootstrap starting.
I20260927 15:16:40.535681 11809 tablet_bootstrap.cc:492] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: Bootstrap starting.
I20260927 15:16:40.536056 11810 tablet_bootstrap.cc:492] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: Bootstrap starting.
I20260927 15:16:40.536376 11808 tablet_bootstrap.cc:654] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.536589 11809 tablet_bootstrap.cc:654] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.536899 11810 tablet_bootstrap.cc:654] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:40.538527 11808 tablet_bootstrap.cc:492] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: No bootstrap required, opened a new log
I20260927 15:16:40.538612 11808 ts_tablet_manager.cc:1397] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:40.538756 11810 tablet_bootstrap.cc:492] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: No bootstrap required, opened a new log
I20260927 15:16:40.538833 11810 ts_tablet_manager.cc:1397] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:40.539044 11808 raft_consensus.cc:348] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.539150 11808 raft_consensus.cc:374] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.539180 11808 raft_consensus.cc:729] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48e15a52d054e0792e4e71ab61a1c3f, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.539211 11810 raft_consensus.cc:348] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.539297 11810 raft_consensus.cc:374] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.539327 11810 raft_consensus.cc:729] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889c030fe6e1425db1923b7489f94159, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.539311 11808 consensus_queue.cc:260] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.539431 11810 consensus_queue.cc:260] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.539558 11808 ts_tablet_manager.cc:1428] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:40.539662 11810 ts_tablet_manager.cc:1428] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:40.539731 11814 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.539848 11814 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:40.539899 11815 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.540001 11815 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:40.542763 11809 tablet_bootstrap.cc:492] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: No bootstrap required, opened a new log
I20260927 15:16:40.542862 11809 ts_tablet_manager.cc:1397] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 15:16:40.543356 11809 raft_consensus.cc:348] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.543479 11809 raft_consensus.cc:374] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:40.543519 11809 raft_consensus.cc:729] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a7be2b2859460c88b132c8db009f2d, State: Initialized, Role: FOLLOWER
I20260927 15:16:40.543666 11809 consensus_queue.cc:260] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.543959 11809 ts_tablet_manager.cc:1428] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:40.543965 11816 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.544174 11816 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:40.731529 11814 raft_consensus.cc:482] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:40.731664 11814 raft_consensus.cc:504] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.732049 11814 leader_election.cc:288] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889c030fe6e1425db1923b7489f94159 (127.9.242.130:44863), f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845)
I20260927 15:16:40.732482 11599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77310e5469ee4ec79a1736a417ea67a7" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889c030fe6e1425db1923b7489f94159" is_pre_election: true
I20260927 15:16:40.732517 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77310e5469ee4ec79a1736a417ea67a7" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a7be2b2859460c88b132c8db009f2d" is_pre_election: true
I20260927 15:16:40.732620 11599 raft_consensus.cc:2454] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 0.
I20260927 15:16:40.732647 11739 raft_consensus.cc:2454] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 0.
I20260927 15:16:40.732915 11392 leader_election.cc:302] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889c030fe6e1425db1923b7489f94159, d48e15a52d054e0792e4e71ab61a1c3f; no voters:
I20260927 15:16:40.733114 11814 raft_consensus.cc:2781] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:40.733183 11814 raft_consensus.cc:482] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:40.733232 11814 raft_consensus.cc:3037] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.734251 11814 raft_consensus.cc:504] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.734555 11814 leader_election.cc:288] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 election: Requested vote from peers 889c030fe6e1425db1923b7489f94159 (127.9.242.130:44863), f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845)
I20260927 15:16:40.734870 11599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77310e5469ee4ec79a1736a417ea67a7" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889c030fe6e1425db1923b7489f94159"
I20260927 15:16:40.734882 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77310e5469ee4ec79a1736a417ea67a7" candidate_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a7be2b2859460c88b132c8db009f2d"
I20260927 15:16:40.734977 11599 raft_consensus.cc:3037] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.734984 11739 raft_consensus.cc:3037] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:40.735808 11739 raft_consensus.cc:2454] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 1.
I20260927 15:16:40.735875 11599 raft_consensus.cc:2454] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d48e15a52d054e0792e4e71ab61a1c3f in term 1.
I20260927 15:16:40.736105 11392 leader_election.cc:302] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d48e15a52d054e0792e4e71ab61a1c3f, f5a7be2b2859460c88b132c8db009f2d; no voters:
I20260927 15:16:40.736245 11814 raft_consensus.cc:2781] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:40.736343 11814 raft_consensus.cc:686] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 LEADER]: Becoming Leader. State: Replica: d48e15a52d054e0792e4e71ab61a1c3f, State: Running, Role: LEADER
I20260927 15:16:40.736478 11814 consensus_queue.cc:237] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } }
I20260927 15:16:40.737062 11827 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: TxnStatusTablet state changed. Reason: New leader d48e15a52d054e0792e4e71ab61a1c3f. Latest consensus state: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.737164 11827 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:16:40.737414 11851 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:16:40.737571 11851 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:16:40.738195 11302 catalog_manager.cc:5495] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d48e15a52d054e0792e4e71ab61a1c3f (127.9.242.129). New cstate: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } health_report { overall_health: UNKNOWN } } }
I20260927 15:16:40.781237 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.777839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:39529 (local address 127.0.0.1:54508)
0927 15:16:40.777983 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.777998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.778094 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.778392 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.778395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.778402 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:40.778492 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.778497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.779482 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.779485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.780081 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.780089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.780616 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.780618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.780683 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.780725 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.780995 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.781002 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.781043 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.781102 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:16:40.781266 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.777948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.9.242.129:39529)
0927 15:16:40.778084 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.778088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.778104 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:16:40.778216 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.778219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.778270 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.778357 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:40.778579 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.779366 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.780179 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.780526 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.780622 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.780847 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.780927 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.780984 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.781054 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.781108 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:40.783635 11739 raft_consensus.cc:1261] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Refusing update from remote peer d48e15a52d054e0792e4e71ab61a1c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:40.783677 11599 raft_consensus.cc:1261] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Refusing update from remote peer d48e15a52d054e0792e4e71ab61a1c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:40.784068 11827 consensus_queue.cc:1035] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:40.784224 11814 consensus_queue.cc:1035] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:40.785497 11815 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: TxnStatusTablet state changed. Reason: New leader d48e15a52d054e0792e4e71ab61a1c3f. Latest consensus state: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.785619 11815 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:40.785624 11816 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: TxnStatusTablet state changed. Reason: New leader d48e15a52d054e0792e4e71ab61a1c3f. Latest consensus state: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.786019 11816 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:40.786806 11814 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.786907 11814 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:16:40.789005 11819 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.789104 11819 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P d48e15a52d054e0792e4e71ab61a1c3f: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:16:40.789640 11815 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.790050 11815 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P 889c030fe6e1425db1923b7489f94159: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:40.791750 11816 tablet_replica.cc:439] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 } } peers { permanent_uuid: "889c030fe6e1425db1923b7489f94159" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44863 } } peers { permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 } } }
I20260927 15:16:40.791865 11816 tablet_replica.cc:442] T 77310e5469ee4ec79a1736a417ea67a7 P f5a7be2b2859460c88b132c8db009f2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:40.791946 11822 mvcc.cc:205] Tried to move back new op lower bound from 7333978934405791744 to 7333978934217777152. Current Snapshot: MvccSnapshot[applied={T|T < 7333978934405791744 or (T in {7333978934405791744})}]
I20260927 15:16:40.796286 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.792885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.9.242.130:44863)
0927 15:16:40.792983 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.792987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.793000 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:16:40.793080 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.793083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.793138 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.793249 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:40.793502 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.794174 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.794960 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.795197 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.795285 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.795520 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.795944 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.796055 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.796109 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.796154 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:16:40.806843 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.803598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:35329 (local address 127.9.242.129:39529)
0927 15:16:40.803714 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.803719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.803732 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:16:40.803775 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.803778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.803819 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.803879 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:40.804437 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.804915 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.805736 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.805966 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.806056 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.806412 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.806477 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.806542 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.806661 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.806713 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:16:40.806865 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.803354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:39529 (local address 127.9.242.130:35329)
0927 15:16:40.803588 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.803602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.803692 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.803892 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.803895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.803902 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:40.804336 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.804343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.805010 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.805013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.805621 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.805628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.806139 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.806143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.806232 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.806293 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.806556 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.806562 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.806647 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.806723 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260927 15:16:40.813411 11738 raft_consensus.cc:1261] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Refusing update from remote peer 889c030fe6e1425db1923b7489f94159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:40.813472 11459 raft_consensus.cc:1261] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Refusing update from remote peer 889c030fe6e1425db1923b7489f94159: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:40.813910 11839 consensus_queue.cc:1035] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a7be2b2859460c88b132c8db009f2d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:40.814271 11815 consensus_queue.cc:1035] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d48e15a52d054e0792e4e71ab61a1c3f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:40.819625 11824 mvcc.cc:205] Tried to move back new op lower bound from 7333978934527500288 to 7333978933186379776. Current Snapshot: MvccSnapshot[applied={T|T < 7333978934527500288}]
I20260927 15:16:40.839128 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.835528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:48351 (local address 127.9.242.130:44863)
0927 15:16:40.835627 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.835630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.835995 (+ 365us) server_negotiation.cc:407] Connection header received
0927 15:16:40.836177 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.836180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.836232 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.836277 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:40.836626 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.837239 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.838118 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.838344 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.838424 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.838701 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.838790 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.838826 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.838969 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.839023 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:16:40.839509 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.835338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44863 (local address 127.9.242.129:48351)
0927 15:16:40.835924 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:40.835938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:40.836065 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:40.836406 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:40.836409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:40.836416 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:40.836511 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.836518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.837344 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.837347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.838019 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:40.838028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.838433 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.838436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:40.838513 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.838553 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:40.838875 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:40.838881 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:16:40.839295 (+ 414us) client_negotiation.cc:240] Negotiation successful
0927 15:16:40.839372 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":102,"threads_started":1}
I20260927 15:16:40.864025 11301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:40.864180 11301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:40.866242 11301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab681f1418fd411ba45d1bf801b08c33: Sending DeleteTablet for 3 replicas of tablet 46ddd825716442cf98fbea696e08a787
I20260927 15:16:40.866760 11439 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ddd825716442cf98fbea696e08a787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:40 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54482
I20260927 15:16:40.866942 11834 tablet_replica.cc:330] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: stopping tablet replica
I20260927 15:16:40.866930 11579 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ddd825716442cf98fbea696e08a787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:40 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:42582
I20260927 15:16:40.867051 11834 raft_consensus.cc:2229] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:40.867069 11832 tablet_replica.cc:330] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: stopping tablet replica
I20260927 15:16:40.867152 11834 raft_consensus.cc:2258] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:40.867154 11832 raft_consensus.cc:2229] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:40.867424 11832 raft_consensus.cc:2258] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:40.868142 11832 ts_tablet_manager.cc:1905] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:40.868142 11834 ts_tablet_manager.cc:1905] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:40.870922 11832 ts_tablet_manager.cc:1918] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:40.870991 11832 log.cc:1196] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/46ddd825716442cf98fbea696e08a787
I20260927 15:16:40.871104 11834 ts_tablet_manager.cc:1918] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:40.871181 11834 log.cc:1196] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/46ddd825716442cf98fbea696e08a787
I20260927 15:16:40.871218 11832 ts_tablet_manager.cc:1939] T 46ddd825716442cf98fbea696e08a787 P 889c030fe6e1425db1923b7489f94159: Deleting consensus metadata
I20260927 15:16:40.871390 11834 ts_tablet_manager.cc:1939] T 46ddd825716442cf98fbea696e08a787 P d48e15a52d054e0792e4e71ab61a1c3f: Deleting consensus metadata
I20260927 15:16:40.871874 11719 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ddd825716442cf98fbea696e08a787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:40 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:38402
I20260927 15:16:40.872072 11833 tablet_replica.cc:330] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: stopping tablet replica
I20260927 15:16:40.872205 11833 raft_consensus.cc:2229] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:40.872326 11833 raft_consensus.cc:2258] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:40.873063 11833 ts_tablet_manager.cc:1905] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:40.873163 11288 catalog_manager.cc:4841] TS 889c030fe6e1425db1923b7489f94159 (127.9.242.130:44863): tablet 46ddd825716442cf98fbea696e08a787 (table test-table [id=c08a4d48063a4c26a56dfca1f81eda40]) successfully deleted
I20260927 15:16:40.873878 11289 catalog_manager.cc:4841] TS d48e15a52d054e0792e4e71ab61a1c3f (127.9.242.129:39529): tablet 46ddd825716442cf98fbea696e08a787 (table test-table [id=c08a4d48063a4c26a56dfca1f81eda40]) successfully deleted
I20260927 15:16:40.875607 11833 ts_tablet_manager.cc:1918] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:40.875667 11833 log.cc:1196] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/46ddd825716442cf98fbea696e08a787
I20260927 15:16:40.875823 11833 ts_tablet_manager.cc:1939] T 46ddd825716442cf98fbea696e08a787 P f5a7be2b2859460c88b132c8db009f2d: Deleting consensus metadata
I20260927 15:16:40.876703 11288 catalog_manager.cc:4841] TS f5a7be2b2859460c88b132c8db009f2d (127.9.242.131:35845): tablet 46ddd825716442cf98fbea696e08a787 (table test-table [id=c08a4d48063a4c26a56dfca1f81eda40]) successfully deleted
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:16:40.885880 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.876766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.9.242.129:39529)
0927 15:16:40.876881 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.876885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.877120 (+ 235us) server_negotiation.cc:407] Connection header received
0927 15:16:40.877223 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.877226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.877268 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.877330 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.878085 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.878584 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.879374 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.879582 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.879664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.882668 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.882680 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.882698 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.882721 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.884224 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.884607 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.884613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.884614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.884689 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.884923 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.884926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.884928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.885167 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.885366 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.885567 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.885715 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 15:16:40.886283 11498 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54520
I20260927 15:16:40.895177 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:40.887856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.9.242.190:35815)
0927 15:16:40.887977 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:40.887982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:40.888361 (+ 379us) server_negotiation.cc:407] Connection header received
0927 15:16:40.888457 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:40.888460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:40.888507 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:40.888571 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:40.889164 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.889749 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.890516 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:40.890736 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:40.890821 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:40.892066 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:40.892070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:40.892073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:40.892098 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:40.893344 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.893926 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.893929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.893930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.894013 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:40.894277 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:40.894281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:40.894282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:40.894476 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:40.894642 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:40.894819 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:16:40.894935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 15:16:40.895748 11301 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47308
I20260927 15:16:40.897269 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 11374
I20260927 15:16:40.905645 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 11514
I20260927 15:16:40.910379 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 11654
I20260927 15:16:40.914760 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 11270
2026-09-27T15:16:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2628 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11872](info): set up 2 sockets
Sep 27 15:16:40 dist-test-slave-m4lt krb5kdc[11872](info): commencing operation
krb5kdc: starting...
W20260927 15:16:41.873325 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:41 dist-test-slave-m4lt krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:41.986856 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:42157
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:42.097487 11885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:42.097807 11885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:42.097872 11885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:42.101892 11885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:42.101949 11885 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:16:42.101972 11885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:42.101996 11885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:42.102026 11885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:42.102051 11885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:42.107358 11885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:42157
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:42.108390 11885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:42.109381 11885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:42.117630 11891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:42.117679 11890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:42.117748 11885 server_base.cc:1034] running on GCE node
W20260927 15:16:42.117702 11893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:42.118440 11885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:42.119513 11885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:42.120678 11885 hybrid_clock.cc:648] HybridClock initialized: now 1790522202120653 us; error 36 us; skew 500 ppm
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:42.123363 11885 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:42.124255 11885 webserver.cc:460] Webserver started at http://127.9.242.190:43765/ using document root <none> and password file <none>
I20260927 15:16:42.124812 11885 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:42.124883 11885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:42.125077 11885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:42.127121 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "67f79503804e426c92f6182039de2777"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "700631fcc632b5b9be83badc42dbd69a"
server_key_iv: "d5d2b874bcc73f62560a4143869be044"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.127673 11885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "67f79503804e426c92f6182039de2777"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "700631fcc632b5b9be83badc42dbd69a"
server_key_iv: "d5d2b874bcc73f62560a4143869be044"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.131397 11885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:16:42.133754 11900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:42.134938 11885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:42.135048 11885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "67f79503804e426c92f6182039de2777"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "700631fcc632b5b9be83badc42dbd69a"
server_key_iv: "d5d2b874bcc73f62560a4143869be044"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.135143 11885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:42.153569 11885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:42.156353 11885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:42.156533 11885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:42.163972 11885 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:42157
I20260927 15:16:42.163978 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:42157 every 8 connection(s)
I20260927 15:16:42.164892 11885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:42.167668 11963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:42.172544 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 11885
I20260927 15:16:42.172639 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:42.172866 10186 external_mini_cluster.cc:1428] Setting key 5a2c1bd6ec189f9394a990f668f1fcb0
I20260927 15:16:42.173087 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Bootstrap starting.
I20260927 15:16:42.175514 11963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:42.176357 11963 log.cc:824] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:42.178637 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: No bootstrap required, opened a new log
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:42.182667 11963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f79503804e426c92f6182039de2777" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42157 } }
I20260927 15:16:42.182897 11963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:42.182952 11963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f79503804e426c92f6182039de2777, State: Initialized, Role: FOLLOWER
I20260927 15:16:42.183430 11963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f79503804e426c92f6182039de2777" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42157 } }
I20260927 15:16:42.183574 11963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:42.183641 11963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:42.183734 11963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:42.184592 11963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f79503804e426c92f6182039de2777" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42157 } }
I20260927 15:16:42.184944 11963 leader_election.cc:302] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67f79503804e426c92f6182039de2777; no voters:
I20260927 15:16:42.185282 11963 leader_election.cc:288] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:42.185385 11968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:42.185593 11968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [term 1 LEADER]: Becoming Leader. State: Replica: 67f79503804e426c92f6182039de2777, State: Running, Role: LEADER
I20260927 15:16:42.185940 11968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f79503804e426c92f6182039de2777" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42157 } }
I20260927 15:16:42.186592 11963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:42.187304 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.174161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.9.242.190:42157)
0927 15:16:42.174590 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.174606 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.174640 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:16:42.175243 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.175260 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.175608 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.175899 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.176775 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.177929 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.179118 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.179604 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.179720 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.182417 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.182422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.182436 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.182462 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.184166 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.184588 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.184593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.184597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.184707 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.184984 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.184987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.184988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.185475 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.185688 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.186029 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.186337 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260927 15:16:42.187485 11970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67f79503804e426c92f6182039de2777. Latest consensus state: current_term: 1 leader_uuid: "67f79503804e426c92f6182039de2777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f79503804e426c92f6182039de2777" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42157 } } }
I20260927 15:16:42.187593 11970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:42.187947 11977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:42.188091 11969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67f79503804e426c92f6182039de2777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f79503804e426c92f6182039de2777" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42157 } } }
I20260927 15:16:42.188169 11969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:42.191150 11977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:42.196513 11977 catalog_manager.cc:1347] Generated new cluster ID: b3be8cc0b7a243798ad8339ed4b5ef4d
I20260927 15:16:42.196573 11977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:42.216688 11977 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:42.218042 11977 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:42.231681 11977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Generated new TSK 0
I20260927 15:16:42.232426 11977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:42.268908 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:42157
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:42.379153 11989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:42.379422 11989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:42.379469 11989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:42.383438 11989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:42.383505 11989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:42.383545 11989 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:16:42.383558 11989 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:16:42.383589 11989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:42.389020 11989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:42.390122 11989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:42.391109 11989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:16:42.400607 11989 server_base.cc:1034] running on GCE node
W20260927 15:16:42.400586 11997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:42.400571 11995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:42.400571 11994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:42.401314 11989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:42.401961 11989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:42.403119 11989 hybrid_clock.cc:648] HybridClock initialized: now 1790522202403100 us; error 32 us; skew 500 ppm
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:42.405725 11989 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:42.406575 11989 webserver.cc:460] Webserver started at http://127.9.242.129:33271/ using document root <none> and password file <none>
I20260927 15:16:42.407099 11989 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:42.407160 11989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:42.407357 11989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:42.409346 11989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "7353980156374f84b85242e2c2934631"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "7955135ad0547f0d97b1c1bca0b5b26b"
server_key_iv: "e0f3e51269315314ad584eccf2718205"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.409893 11989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "7353980156374f84b85242e2c2934631"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "7955135ad0547f0d97b1c1bca0b5b26b"
server_key_iv: "e0f3e51269315314ad584eccf2718205"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.413441 11989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:16:42.415802 12004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:42.416971 11989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:42.417074 11989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "7353980156374f84b85242e2c2934631"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "7955135ad0547f0d97b1c1bca0b5b26b"
server_key_iv: "e0f3e51269315314ad584eccf2718205"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.417171 11989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:42.429028 11989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:42.431749 11989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:42.431918 11989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:42.433312 11989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:42.433360 11989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:42.433413 11989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:42.433442 11989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:42.443244 11989 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:41503
I20260927 15:16:42.443277 12119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:41503 every 8 connection(s)
I20260927 15:16:42.444262 11989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:42.444821 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 11989
I20260927 15:16:42.444937 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:42.445246 10186 external_mini_cluster.cc:1428] Setting key 537f3970fa7e5527bd9beb968a9f9841
I20260927 15:16:42.446931 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.434213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:59791 (local address 127.9.242.190:42157)
0927 15:16:42.434363 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.434368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.435260 (+ 892us) server_negotiation.cc:407] Connection header received
0927 15:16:42.436430 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.436434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.436492 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.436582 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.437927 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.438642 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.439902 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.440156 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.440242 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.443149 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.443153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.443156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.443184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.444546 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.445350 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.445354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.445356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.445445 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.445837 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.445841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.445842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.446045 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.446125 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.446689 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.446811 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:16:42.447948 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.434609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42157 (local address 127.9.242.129:59791)
0927 15:16:42.435084 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.435139 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.436170 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.436790 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.436799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.437166 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:42.437778 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.437793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.438748 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.438751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.439799 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.439811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.440310 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.440314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.440472 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.441524 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:42.441540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:42.442982 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:42.444671 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.444676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.444691 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.445240 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.445540 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.445543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.445545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.445752 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.446147 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:42.446154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:42.446437 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.446635 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.446971 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260927 15:16:42.449340 12120 heartbeater.cc:347] Connected to a master server at 127.9.242.190:42157
I20260927 15:16:42.449486 12120 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:42.450161 12120 heartbeater.cc:511] Master 127.9.242.190:42157 requested a full tablet report, sending...
I20260927 15:16:42.451853 11916 ts_manager.cc:194] Registered new tserver with Master: 7353980156374f84b85242e2c2934631 (127.9.242.129:41503)
I20260927 15:16:42.453419 11916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:59791
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:16:42.460537 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.451412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42157 (local address 127.9.242.129:51097)
0927 15:16:42.451653 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.451665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.451737 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.454449 (+ 2712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.454452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.454662 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:42.454774 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.454781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.455664 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.455667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.456260 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.456268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.456658 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.456660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.456728 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.457172 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:42.457186 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:42.457518 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:42.459256 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.459258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.459260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.459442 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.459701 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.459702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.459704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.459770 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.460119 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:42.460121 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:42.460193 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.460264 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.460381 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260927 15:16:42.460579 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.454106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:51097 (local address 127.9.242.190:42157)
0927 15:16:42.454222 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.454226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.454239 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:16:42.454276 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.454279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.454321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.454398 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.454901 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.455535 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.456367 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.456572 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.456648 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.457642 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.457646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.457649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.457694 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.459146 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.459533 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.459536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.459538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.459624 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.459844 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.459847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.459849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.460043 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.460135 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.460329 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.460433 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:42.472644 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:42157
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:42.577262 12129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:42.577519 12129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:42.577565 12129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:42.581492 12129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:42.581561 12129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:42.581600 12129 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:16:42.581612 12129 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:16:42.581645 12129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:42.586856 12129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:42.587949 12129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:42.588970 12129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:42.598165 12137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:42.598381 12129 server_base.cc:1034] running on GCE node
W20260927 15:16:42.598333 12134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:42.598153 12135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:42.598863 12129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:42.599452 12129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:42.600600 12129 hybrid_clock.cc:648] HybridClock initialized: now 1790522202600574 us; error 45 us; skew 500 ppm
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:42.603149 12129 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:42.603957 12129 webserver.cc:460] Webserver started at http://127.9.242.130:34023/ using document root <none> and password file <none>
I20260927 15:16:42.604473 12129 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:42.604525 12129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:42.604683 12129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:42.606659 12129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "a2fe77618ea243a8a25e965fe097f225"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "c8815caba47d3d6373bf4e4b345f308f"
server_key_iv: "8578b46f87fe0b2be90a7af82651e68c"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.607175 12129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "a2fe77618ea243a8a25e965fe097f225"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "c8815caba47d3d6373bf4e4b345f308f"
server_key_iv: "8578b46f87fe0b2be90a7af82651e68c"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.610740 12129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:42.613006 12144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:42.614260 12129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:42.614353 12129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "a2fe77618ea243a8a25e965fe097f225"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "c8815caba47d3d6373bf4e4b345f308f"
server_key_iv: "8578b46f87fe0b2be90a7af82651e68c"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.614456 12129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:42.629703 12129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:42.632228 12129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:42.632400 12129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:42.633849 12129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:42.633896 12129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:42.633936 12129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:42.633963 12129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:42.643213 12129 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:40737
I20260927 15:16:42.643270 12259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:40737 every 8 connection(s)
I20260927 15:16:42.644186 12129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:42.647449 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.634454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:45617 (local address 127.9.242.190:42157)
0927 15:16:42.634602 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.634608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.635895 (+ 1287us) server_negotiation.cc:407] Connection header received
0927 15:16:42.636967 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.636971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.637024 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.637067 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.638472 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.639178 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.640729 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.641002 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.641097 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.643932 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.643935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.643937 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.643961 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.645425 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.645984 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.645988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.645990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.646069 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.646409 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.646411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.646413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.646585 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.646665 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.647156 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.647286 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:16:42.648041 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 12129
I20260927 15:16:42.648155 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:42.648357 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.635300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42157 (local address 127.9.242.130:45617)
0927 15:16:42.635745 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.635796 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.636737 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.637425 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.637432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.637861 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:42.638331 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.638344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.639299 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.639302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.640615 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.640625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.641148 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.641152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.641347 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.642044 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:42.642065 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:42.643764 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:42.645534 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.645540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.645554 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.645907 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.646172 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.646175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.646177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.646308 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.646682 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:42.646688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:42.646927 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.647078 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.647385 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260927 15:16:42.648452 10186 external_mini_cluster.cc:1428] Setting key e2ab76818e571749599564611e751aa5
I20260927 15:16:42.649678 12260 heartbeater.cc:347] Connected to a master server at 127.9.242.190:42157
I20260927 15:16:42.649828 12260 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:42.650373 12260 heartbeater.cc:511] Master 127.9.242.190:42157 requested a full tablet report, sending...
I20260927 15:16:42.651306 11915 ts_manager.cc:194] Registered new tserver with Master: a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737)
I20260927 15:16:42.652057 11915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:45617
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:16:42.659130 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.651801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42157 (local address 127.9.242.130:36075)
0927 15:16:42.652174 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.652189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.652288 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.652506 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.652509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.652677 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:42.652783 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.652788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.653824 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.653827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.654612 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.654622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.655052 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.655056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.655167 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.655652 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:42.655668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:42.656095 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:42.657636 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.657640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.657642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.657904 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.658188 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.658191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.658193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.658280 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.658633 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:42.658636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:42.658719 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.658861 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.658990 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":98,"threads_started":1}
I20260927 15:16:42.659139 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.651859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:36075 (local address 127.9.242.190:42157)
0927 15:16:42.651967 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.651971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.652246 (+ 275us) server_negotiation.cc:407] Connection header received
0927 15:16:42.652371 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.652374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.652417 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.652495 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.652883 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.653635 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.654715 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.654935 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.655016 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.656237 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.656240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.656242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.656264 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.657525 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.657993 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.657997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.657999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.658084 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.658348 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.658350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.658351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.658545 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.658617 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.658909 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.659022 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:42.676033 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:42157
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:42.786248 12269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:42.786512 12269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:42.786558 12269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:42.790683 12269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:42.790766 12269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:42.790825 12269 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 15:16:42.790846 12269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:16:42.790879 12269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:42.796399 12269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:42.797415 12269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:42.798444 12269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:16:42.807770 12269 server_base.cc:1034] running on GCE node
W20260927 15:16:42.807718 12275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:42.807724 12274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:42.808007 12277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:42.808362 12269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:42.808936 12269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:42.810083 12269 hybrid_clock.cc:648] HybridClock initialized: now 1790522202810067 us; error 32 us; skew 500 ppm
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:42.812633 12269 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:42.813426 12269 webserver.cc:460] Webserver started at http://127.9.242.131:35697/ using document root <none> and password file <none>
I20260927 15:16:42.813977 12269 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:42.814034 12269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:42.814198 12269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:42.816100 12269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "233a42c709544ce19b0a3accada2e51b"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "3ea112ad4a33732c42fe23f4d499086b"
server_key_iv: "a718c870facf01d77a7aae500f9524c3"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.816576 12269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "233a42c709544ce19b0a3accada2e51b"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "3ea112ad4a33732c42fe23f4d499086b"
server_key_iv: "a718c870facf01d77a7aae500f9524c3"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.820183 12269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:16:42.822584 12284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:42.823840 12269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:42.823940 12269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "233a42c709544ce19b0a3accada2e51b"
format_stamp: "Formatted at 2026-09-27 15:16:42 on dist-test-slave-m4lt"
server_key: "3ea112ad4a33732c42fe23f4d499086b"
server_key_iv: "a718c870facf01d77a7aae500f9524c3"
server_key_version: "encryptionkey@0"
I20260927 15:16:42.824064 12269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:42.854851 12269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:42.858108 12269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:42.858392 12269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:42.859906 12269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:42.859953 12269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:42.859992 12269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:42.860019 12269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:42.870167 12269 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:42715
I20260927 15:16:42.870262 12399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:42715 every 8 connection(s)
I20260927 15:16:42.871093 12269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:42.871989 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 12269
I20260927 15:16:42.872083 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:42.872411 10186 external_mini_cluster.cc:1428] Setting key 148b38876019590668d409defeb32241
I20260927 15:16:42.874364 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.860749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:53419 (local address 127.9.242.190:42157)
0927 15:16:42.860866 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.860869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.861997 (+ 1128us) server_negotiation.cc:407] Connection header received
0927 15:16:42.863000 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.863003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.863059 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.863141 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.864740 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.865554 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.867279 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.867492 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.867577 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.870603 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.870606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.870608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.870633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.872125 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.872707 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.872711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.872713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.872798 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.873190 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.873194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.873195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.873408 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.873483 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.874086 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.874211 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:16:42.875125 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.861305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42157 (local address 127.9.242.131:53419)
0927 15:16:42.861844 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.861887 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.862771 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.863358 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.863368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.863865 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:42.864585 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.864606 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.865711 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.865714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.867139 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.867151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.867658 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.867663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.867862 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.868546 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:42.868565 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:42.870439 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:42.872236 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.872242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.872258 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.872586 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.872903 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.872905 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.872907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.873084 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.873511 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:42.873518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:42.873827 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.873993 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.874261 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":134,"threads_started":1}
I20260927 15:16:42.876197 12400 heartbeater.cc:347] Connected to a master server at 127.9.242.190:42157
I20260927 15:16:42.876350 12400 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:42.876832 12400 heartbeater.cc:511] Master 127.9.242.190:42157 requested a full tablet report, sending...
I20260927 15:16:42.877972 11917 ts_manager.cc:194] Registered new tserver with Master: 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715)
I20260927 15:16:42.878677 11917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:53419
I20260927 15:16:42.885433 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.878529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:34493 (local address 127.9.242.190:42157)
0927 15:16:42.878651 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.878656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.878870 (+ 214us) server_negotiation.cc:407] Connection header received
0927 15:16:42.879002 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.879005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.879047 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.879103 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.879529 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.880249 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.881024 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.881291 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.881369 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.882486 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.882489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.882491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.882510 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.883777 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.884254 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.884257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.884258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.884329 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.884637 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.884639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.884640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.884802 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.884875 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.885197 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.885295 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:16:42.885450 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.878441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42157 (local address 127.9.242.131:34493)
0927 15:16:42.878789 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.878802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.878904 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.879144 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.879148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.879330 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:42.879425 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.879431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.880374 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.880377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.880926 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.880933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.881395 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.881398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.881474 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.881971 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:42.881982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:42.882371 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:42.883882 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.883885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.883886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.884092 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.884470 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:42.884473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:42.884474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:42.884557 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:42.884907 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:42.884910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:42.885010 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.885129 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.885274 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":94,"threads_started":1}
I20260927 15:16:42.886694 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:16:42 dist-test-slave-m4lt krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:42.910171 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.902738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.9.242.190:42157)
0927 15:16:42.902857 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.902860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.902872 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:16:42.902912 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.902914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.902951 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.903016 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:42.903675 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.904202 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.905073 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.905285 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.905361 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.907681 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:42.907684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:42.907686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:42.907710 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:42.908848 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.909227 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.909229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.909231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.909300 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:42.909484 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:42.909487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:42.909488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:42.909643 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:42.909782 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.909924 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.910012 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:16:42.913000 11917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:42.916080 11917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:42.937959 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.924789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.0.0.1:34152)
0927 15:16:42.925987 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.926065 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.926289 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.933493 (+ 7204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.933499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.933524 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:42.933716 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.933724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.935144 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.935148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.936526 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.936535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.937418 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.937421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.937527 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.937565 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.937689 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.937812 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":67,"threads_started":1}
I20260927 15:16:42.937969 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.932517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.9.242.129:41503)
0927 15:16:42.932938 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.932959 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.932984 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:16:42.933067 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.933073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.933317 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.933478 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:42.933854 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.934945 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.936664 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.937303 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.937395 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.937571 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.937725 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.937811 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":100,"threads_started":1}
I20260927 15:16:42.940446 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.925111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:42715 (local address 127.0.0.1:44678)
0927 15:16:42.933102 (+ 7991us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.933116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.933215 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.935842 (+ 2627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.935845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.935863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:42.935957 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.935963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.937295 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.937299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.938688 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.938705 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.940059 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.940062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.940153 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.940174 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.940246 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.940309 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7919,"mutex_wait_us":21,"thread_start_us":78,"threads_started":1}
I20260927 15:16:42.940478 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.932657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.9.242.131:42715)
0927 15:16:42.935172 (+ 2515us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.935185 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.935209 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:16:42.935286 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.935292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.935524 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.935681 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:42.936078 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.937159 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.938803 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.939956 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.940050 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.940198 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.940280 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.940363 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2396,"thread_start_us":127,"threads_started":1}
I20260927 15:16:42.940793 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.925480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40737 (local address 127.0.0.1:48836)
0927 15:16:42.933800 (+ 8320us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:42.933814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:42.933882 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:42.936663 (+ 2781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:42.936666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:42.936684 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:42.936780 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.936785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.938109 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.938113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.939213 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:42.939223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.940423 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.940426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:42.940517 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.940538 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:16:42.940600 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:16:42.940660 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8259,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260927 15:16:42.941226 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:42.932616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.9.242.130:40737)
0927 15:16:42.936225 (+ 3609us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:42.936239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:42.936261 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:16:42.936329 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:42.936334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:42.936521 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:42.936661 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:42.936889 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.937985 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.939386 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:42.940120 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:42.940229 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:42.940365 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:42.940996 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 15:16:42.941084 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3507,"thread_start_us":156,"threads_started":1}
I20260927 15:16:42.941182 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 46109cd8ec7b440990cb65d8eae81b1b (DEFAULT_TABLE table=test-table [id=cd9955d313c2484b828e269a2cc074db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:42.942298 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46109cd8ec7b440990cb65d8eae81b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:42.943439 12194 tablet_service.cc:1467] Processing CreateTablet for tablet 46109cd8ec7b440990cb65d8eae81b1b (DEFAULT_TABLE table=test-table [id=cd9955d313c2484b828e269a2cc074db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:42.944455 12194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46109cd8ec7b440990cb65d8eae81b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:42.948365 12334 tablet_service.cc:1467] Processing CreateTablet for tablet 46109cd8ec7b440990cb65d8eae81b1b (DEFAULT_TABLE table=test-table [id=cd9955d313c2484b828e269a2cc074db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:42.949473 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46109cd8ec7b440990cb65d8eae81b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:42.950489 12423 tablet_bootstrap.cc:492] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Bootstrap starting.
I20260927 15:16:42.951305 12424 tablet_bootstrap.cc:492] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Bootstrap starting.
I20260927 15:16:42.952721 12423 tablet_bootstrap.cc:654] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:42.953316 12424 tablet_bootstrap.cc:654] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:42.953552 12423 log.cc:824] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:42.954090 12424 log.cc:824] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:42.955482 12423 tablet_bootstrap.cc:492] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: No bootstrap required, opened a new log
I20260927 15:16:42.955664 12423 ts_tablet_manager.cc:1397] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 15:16:42.955888 12424 tablet_bootstrap.cc:492] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: No bootstrap required, opened a new log
I20260927 15:16:42.956053 12424 ts_tablet_manager.cc:1397] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:42.957944 12427 tablet_bootstrap.cc:492] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Bootstrap starting.
I20260927 15:16:42.959605 12427 tablet_bootstrap.cc:654] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:42.959538 12424 raft_consensus.cc:348] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:42.959514 12423 raft_consensus.cc:348] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:42.959772 12424 raft_consensus.cc:374] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:42.959772 12423 raft_consensus.cc:374] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:42.959825 12423 raft_consensus.cc:729] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7353980156374f84b85242e2c2934631, State: Initialized, Role: FOLLOWER
I20260927 15:16:42.959825 12424 raft_consensus.cc:729] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fe77618ea243a8a25e965fe097f225, State: Initialized, Role: FOLLOWER
I20260927 15:16:42.960259 12423 consensus_queue.cc:260] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:42.960263 12424 consensus_queue.cc:260] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:42.960392 12427 log.cc:824] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:42.961078 12260 heartbeater.cc:503] Master 127.9.242.190:42157 was elected leader, sending a full tablet report...
I20260927 15:16:42.961108 12120 heartbeater.cc:503] Master 127.9.242.190:42157 was elected leader, sending a full tablet report...
I20260927 15:16:42.961206 12424 ts_tablet_manager.cc:1428] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:16:42.962148 12423 ts_tablet_manager.cc:1428] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 15:16:42.962471 12427 tablet_bootstrap.cc:492] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: No bootstrap required, opened a new log
I20260927 15:16:42.962633 12427 ts_tablet_manager.cc:1397] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:16:42.966358 12427 raft_consensus.cc:348] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:42.966583 12427 raft_consensus.cc:374] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:42.966640 12427 raft_consensus.cc:729] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233a42c709544ce19b0a3accada2e51b, State: Initialized, Role: FOLLOWER
I20260927 15:16:42.967020 12427 consensus_queue.cc:260] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:42.967597 12400 heartbeater.cc:503] Master 127.9.242.190:42157 was elected leader, sending a full tablet report...
I20260927 15:16:42.967836 12427 ts_tablet_manager.cc:1428] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 15:16:43.121989 12401 tablet.cc:2369] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:43.145607 12261 tablet.cc:2369] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:43.195935 12121 tablet.cc:2369] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:43.342435 12429 raft_consensus.cc:482] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:43.342679 12429 raft_consensus.cc:504] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.343825 12429 leader_election.cc:288] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7353980156374f84b85242e2c2934631 (127.9.242.129:41503), 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715)
I20260927 15:16:43.348922 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.344180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:54975 (local address 127.9.242.129:41503)
0927 15:16:43.344317 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.344322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.344600 (+ 278us) server_negotiation.cc:407] Connection header received
0927 15:16:43.344785 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.344788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.344831 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.344899 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:43.345103 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.345890 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.347655 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.348262 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.348382 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.348433 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.348712 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.348806 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:16:43.348915 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.344296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:48829 (local address 127.9.242.131:42715)
0927 15:16:43.344418 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.344422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.344508 (+ 86us) server_negotiation.cc:407] Connection header received
0927 15:16:43.344680 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.344683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.344723 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.344795 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:43.345042 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.345878 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.347666 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.348268 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.348382 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.348433 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.348712 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.348783 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:16:43.349009 12433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.344168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.9.242.130:54975)
0927 15:16:43.344499 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.344512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.344627 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.344921 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.344923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.344936 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:43.345005 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.345011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.345998 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.346002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.347525 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.347535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.348400 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.348404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.348566 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.348616 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.348694 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.348817 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":193,"thread_start_us":75,"threads_started":1}
I20260927 15:16:43.349017 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.344094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:42715 (local address 127.9.242.130:48829)
0927 15:16:43.344416 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.344429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.344547 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.344806 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.344810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.344824 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:43.344938 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.344945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.345998 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.346002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.347524 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.347535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.348400 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.348404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.348575 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.348611 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.348694 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.348820 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":217,"thread_start_us":86,"threads_started":1}
I20260927 15:16:43.349884 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46109cd8ec7b440990cb65d8eae81b1b" candidate_uuid: "a2fe77618ea243a8a25e965fe097f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233a42c709544ce19b0a3accada2e51b" is_pre_election: true
I20260927 15:16:43.349879 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46109cd8ec7b440990cb65d8eae81b1b" candidate_uuid: "a2fe77618ea243a8a25e965fe097f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7353980156374f84b85242e2c2934631" is_pre_election: true
I20260927 15:16:43.350203 12074 raft_consensus.cc:2454] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2fe77618ea243a8a25e965fe097f225 in term 0.
I20260927 15:16:43.350203 12354 raft_consensus.cc:2454] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2fe77618ea243a8a25e965fe097f225 in term 0.
I20260927 15:16:43.350704 12148 leader_election.cc:302] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 233a42c709544ce19b0a3accada2e51b, 7353980156374f84b85242e2c2934631, a2fe77618ea243a8a25e965fe097f225; no voters:
I20260927 15:16:43.350937 12429 raft_consensus.cc:2781] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:43.351014 12429 raft_consensus.cc:482] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:43.351059 12429 raft_consensus.cc:3037] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.352044 12429 raft_consensus.cc:504] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.352373 12429 leader_election.cc:288] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [CANDIDATE]: Term 1 election: Requested vote from peers 7353980156374f84b85242e2c2934631 (127.9.242.129:41503), 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715)
I20260927 15:16:43.352684 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46109cd8ec7b440990cb65d8eae81b1b" candidate_uuid: "a2fe77618ea243a8a25e965fe097f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233a42c709544ce19b0a3accada2e51b"
I20260927 15:16:43.352705 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46109cd8ec7b440990cb65d8eae81b1b" candidate_uuid: "a2fe77618ea243a8a25e965fe097f225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7353980156374f84b85242e2c2934631"
I20260927 15:16:43.352787 12354 raft_consensus.cc:3037] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.352809 12074 raft_consensus.cc:3037] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.353790 12074 raft_consensus.cc:2454] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2fe77618ea243a8a25e965fe097f225 in term 1.
I20260927 15:16:43.353833 12354 raft_consensus.cc:2454] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2fe77618ea243a8a25e965fe097f225 in term 1.
I20260927 15:16:43.354092 12147 leader_election.cc:302] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7353980156374f84b85242e2c2934631, a2fe77618ea243a8a25e965fe097f225; no voters:
I20260927 15:16:43.354251 12429 raft_consensus.cc:2781] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:43.354416 12429 raft_consensus.cc:686] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 1 LEADER]: Becoming Leader. State: Replica: a2fe77618ea243a8a25e965fe097f225, State: Running, Role: LEADER
I20260927 15:16:43.354730 12429 consensus_queue.cc:237] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.357981 11917 catalog_manager.cc:5495] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2fe77618ea243a8a25e965fe097f225 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "a2fe77618ea243a8a25e965fe097f225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:16:43 dist-test-slave-m4lt krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522202, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:43.403291 11966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.395448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.9.242.190:42157)
0927 15:16:43.395580 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.395584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.395806 (+ 222us) server_negotiation.cc:407] Connection header received
0927 15:16:43.395925 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.395927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.395965 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.396037 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:43.396702 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.397205 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.398193 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.398420 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.398493 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.400588 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:43.400590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:43.400593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:43.400613 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:43.401823 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:43.402274 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:43.402277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:43.402278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:43.402345 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:43.402563 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:43.402565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:43.402566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:43.402716 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:43.402782 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.403033 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.403116 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:16:43.403311 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.395373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42157 (local address 127.0.0.1:43550)
0927 15:16:43.395721 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.395738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.395841 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.396082 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.396084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.396482 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:43.396604 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.396610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.397342 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.397345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.398025 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.398036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.398523 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.398526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.398608 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.399158 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:43.399170 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:43.400477 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:43.401952 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:43.401960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:43.401965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:43.402194 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:43.402418 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:43.402421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:43.402422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:43.402501 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:43.402800 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:43.402806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:43.402906 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.402994 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.403166 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":141,"threads_started":1}
I20260927 15:16:43.407088 11917 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:43550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:16:43.407753 11917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:43.412312 12194 tablet_service.cc:1467] Processing CreateTablet for tablet 09b0ddeee73f4372ae4dbdafa0ce7054 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4df06fac3e041a68c4344f8d4402a3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:16:43.412293 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 09b0ddeee73f4372ae4dbdafa0ce7054 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4df06fac3e041a68c4344f8d4402a3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:16:43.412334 12334 tablet_service.cc:1467] Processing CreateTablet for tablet 09b0ddeee73f4372ae4dbdafa0ce7054 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4df06fac3e041a68c4344f8d4402a3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:16:43.413003 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b0ddeee73f4372ae4dbdafa0ce7054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:43.413004 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b0ddeee73f4372ae4dbdafa0ce7054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:43.413009 12194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b0ddeee73f4372ae4dbdafa0ce7054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:43.415648 12424 tablet_bootstrap.cc:492] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: Bootstrap starting.
I20260927 15:16:43.415784 12423 tablet_bootstrap.cc:492] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: Bootstrap starting.
I20260927 15:16:43.415851 12427 tablet_bootstrap.cc:492] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: Bootstrap starting.
I20260927 15:16:43.416687 12423 tablet_bootstrap.cc:654] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:43.416736 12427 tablet_bootstrap.cc:654] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:43.416972 12424 tablet_bootstrap.cc:654] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:43.417865 12423 tablet_bootstrap.cc:492] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: No bootstrap required, opened a new log
I20260927 15:16:43.417939 12427 tablet_bootstrap.cc:492] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: No bootstrap required, opened a new log
I20260927 15:16:43.417951 12423 ts_tablet_manager.cc:1397] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 15:16:43.418018 12427 ts_tablet_manager.cc:1397] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:43.418371 12427 raft_consensus.cc:348] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.418421 12423 raft_consensus.cc:348] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.418481 12427 raft_consensus.cc:374] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:43.418535 12423 raft_consensus.cc:374] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:43.418535 12427 raft_consensus.cc:729] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233a42c709544ce19b0a3accada2e51b, State: Initialized, Role: FOLLOWER
I20260927 15:16:43.418579 12423 raft_consensus.cc:729] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7353980156374f84b85242e2c2934631, State: Initialized, Role: FOLLOWER
I20260927 15:16:43.418671 12427 consensus_queue.cc:260] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.418704 12423 consensus_queue.cc:260] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.418927 12427 ts_tablet_manager.cc:1428] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:43.418960 12423 ts_tablet_manager.cc:1428] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:16:43.418970 12428 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.419145 12428 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:43.419507 12431 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.419646 12431 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:43.420773 12424 tablet_bootstrap.cc:492] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: No bootstrap required, opened a new log
I20260927 15:16:43.420872 12424 ts_tablet_manager.cc:1397] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:16:43.421274 12424 raft_consensus.cc:348] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.421365 12424 raft_consensus.cc:374] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:43.421393 12424 raft_consensus.cc:729] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fe77618ea243a8a25e965fe097f225, State: Initialized, Role: FOLLOWER
I20260927 15:16:43.421566 12424 consensus_queue.cc:260] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.421845 12429 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.421974 12429 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:43.422174 12424 ts_tablet_manager.cc:1428] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:16:43.643963 12428 raft_consensus.cc:482] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:43.644186 12428 raft_consensus.cc:504] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.645319 12428 leader_election.cc:288] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737), 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715)
I20260927 15:16:43.650080 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.645707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:45663 (local address 127.9.242.131:42715)
0927 15:16:43.645831 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.645836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.646254 (+ 418us) server_negotiation.cc:407] Connection header received
0927 15:16:43.646387 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.646390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.646432 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.646490 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:43.646921 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.647404 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.648933 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.649357 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.649460 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.649513 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.649860 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.649941 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:16:43.650113 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.645563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40737 (local address 127.9.242.129:50807)
0927 15:16:43.645912 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.645926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.646022 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.646314 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.646318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.646336 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:43.646440 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.646446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.647446 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.647450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.648721 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.648731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.649585 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.649589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.649752 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.649770 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.649828 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.649921 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":27,"thread_start_us":91,"threads_started":1}
I20260927 15:16:43.650425 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.645773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:50807 (local address 127.9.242.130:40737)
0927 15:16:43.645991 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.646004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.646020 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:16:43.646109 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.646112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.646161 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.646247 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:43.646553 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.647336 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.648832 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.649470 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.649571 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.649626 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.650246 (+ 620us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.650316 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260927 15:16:43.650120 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.645564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:42715 (local address 127.9.242.129:45663)
0927 15:16:43.646153 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.646166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.646258 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.646726 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.646728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.646743 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:43.646812 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.646816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.647537 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.647540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.648794 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.648801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.649500 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.649504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.649596 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.649753 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.649828 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.649921 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":566,"thread_start_us":61,"threads_started":1}
I20260927 15:16:43.650998 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b0ddeee73f4372ae4dbdafa0ce7054" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233a42c709544ce19b0a3accada2e51b" is_pre_election: true
I20260927 15:16:43.651034 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b0ddeee73f4372ae4dbdafa0ce7054" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225" is_pre_election: true
I20260927 15:16:43.651146 12354 raft_consensus.cc:2454] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 0.
I20260927 15:16:43.651242 12214 raft_consensus.cc:2454] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 0.
I20260927 15:16:43.651460 12008 leader_election.cc:302] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233a42c709544ce19b0a3accada2e51b, 7353980156374f84b85242e2c2934631; no voters:
I20260927 15:16:43.651739 12428 raft_consensus.cc:2781] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:43.651803 12428 raft_consensus.cc:482] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:43.651844 12428 raft_consensus.cc:3037] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.652765 12428 raft_consensus.cc:504] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.653100 12428 leader_election.cc:288] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 election: Requested vote from peers a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737), 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715)
I20260927 15:16:43.653388 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b0ddeee73f4372ae4dbdafa0ce7054" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233a42c709544ce19b0a3accada2e51b"
I20260927 15:16:43.653400 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b0ddeee73f4372ae4dbdafa0ce7054" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225"
I20260927 15:16:43.653499 12214 raft_consensus.cc:3037] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.653501 12354 raft_consensus.cc:3037] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.654404 12354 raft_consensus.cc:2454] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 1.
I20260927 15:16:43.654526 12214 raft_consensus.cc:2454] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 1.
I20260927 15:16:43.654665 12008 leader_election.cc:302] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233a42c709544ce19b0a3accada2e51b, 7353980156374f84b85242e2c2934631; no voters:
I20260927 15:16:43.654812 12428 raft_consensus.cc:2781] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:43.655025 12428 raft_consensus.cc:686] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [term 1 LEADER]: Becoming Leader. State: Replica: 7353980156374f84b85242e2c2934631, State: Running, Role: LEADER
I20260927 15:16:43.655347 12428 consensus_queue.cc:237] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.657368 12448 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: TxnStatusTablet state changed. Reason: New leader 7353980156374f84b85242e2c2934631. Latest consensus state: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.657455 12448 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:16:43.657742 12450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:16:43.657924 12450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:16:43.658731 11915 catalog_manager.cc:5495] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7353980156374f84b85242e2c2934631 (127.9.242.129). New cstate: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } health_report { overall_health: UNKNOWN } } }
I20260927 15:16:43.714789 12441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.710891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.0.0.1:34154)
0927 15:16:43.711040 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.711056 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.711151 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.711438 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.711440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.711446 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:43.711533 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.711537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.712514 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.712516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.713105 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.713112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.713668 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.713672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.713758 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.713801 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:43.714546 (+ 745us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:43.714554 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.714590 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.714659 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:16:43.714797 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.711025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.9.242.129:41503)
0927 15:16:43.711158 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.711163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.711179 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:16:43.711260 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.711263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.711332 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.711431 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:43.711633 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.712416 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.713210 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.713557 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.713679 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.713934 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:43.714464 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:43.714575 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.714608 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.714666 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:16:43.720311 12214 raft_consensus.cc:1261] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Refusing update from remote peer 7353980156374f84b85242e2c2934631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:43.720428 12354 raft_consensus.cc:1261] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Refusing update from remote peer 7353980156374f84b85242e2c2934631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:43.720968 12428 consensus_queue.cc:1035] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:43.721145 12448 consensus_queue.cc:1035] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:43.723990 12429 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: TxnStatusTablet state changed. Reason: New leader 7353980156374f84b85242e2c2934631. Latest consensus state: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.724128 12429 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:43.724483 12431 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: TxnStatusTablet state changed. Reason: New leader 7353980156374f84b85242e2c2934631. Latest consensus state: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.724589 12431 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:43.727243 12448 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.727355 12448 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:16:43.727481 12453 mvcc.cc:205] Tried to move back new op lower bound from 7333978946432016384 to 7333978946174758912. Current Snapshot: MvccSnapshot[applied={T|T < 7333978946432016384}]
I20260927 15:16:43.728401 12431 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.728515 12431 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 233a42c709544ce19b0a3accada2e51b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:43.730947 12455 mvcc.cc:205] Tried to move back new op lower bound from 7333978946432016384 to 7333978946174758912. Current Snapshot: MvccSnapshot[applied={T|T < 7333978946432016384}]
I20260927 15:16:43.732215 12428 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.732333 12428 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P 7353980156374f84b85242e2c2934631: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:16:43.734102 12429 tablet_replica.cc:439] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } } }
I20260927 15:16:43.734225 12429 tablet_replica.cc:442] T 09b0ddeee73f4372ae4dbdafa0ce7054 P a2fe77618ea243a8a25e965fe097f225: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:16:43.738170 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.732874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.9.242.130:40737)
0927 15:16:43.732991 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.732995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.734162 (+ 1167us) server_negotiation.cc:407] Connection header received
0927 15:16:43.734385 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.734389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.734442 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.734513 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:43.734903 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.735615 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.736581 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.736923 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.737036 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.737232 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:43.737785 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:43.737918 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.737969 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.738027 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:16:43.749081 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.745460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:60717 (local address 127.9.242.129:41503)
0927 15:16:43.745596 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.745601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.745892 (+ 291us) server_negotiation.cc:407] Connection header received
0927 15:16:43.746004 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.746006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.746049 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.746106 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:43.746357 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.746851 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.748069 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.748280 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.748352 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.748654 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:43.748749 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:43.748810 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.748906 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.748950 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:16:43.749130 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.745369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.9.242.130:60717)
0927 15:16:43.745807 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.745821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.745923 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.746142 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.746145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.746152 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:43.746274 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.746279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.746953 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.746957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.747956 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.747967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.748376 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.748380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.748474 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.748521 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:43.748815 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:43.748821 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.748895 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.748971 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":95,"threads_started":1}
I20260927 15:16:43.755589 12354 raft_consensus.cc:1261] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Refusing update from remote peer a2fe77618ea243a8a25e965fe097f225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:43.755918 12074 raft_consensus.cc:1261] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Refusing update from remote peer a2fe77618ea243a8a25e965fe097f225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:43.756040 12434 consensus_queue.cc:1035] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:43.756629 12434 consensus_queue.cc:1035] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:43.759660 12454 mvcc.cc:205] Tried to move back new op lower bound from 7333978946578509824 to 7333978944943509504. Current Snapshot: MvccSnapshot[applied={T|T < 7333978946578509824}]
I20260927 15:16:43.781391 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.776543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:41531 (local address 127.9.242.130:40737)
0927 15:16:43.776667 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:43.776671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:43.777949 (+ 1278us) server_negotiation.cc:407] Connection header received
0927 15:16:43.778316 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:43.778319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:43.778373 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:43.778456 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:43.778706 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.779317 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.780215 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.780550 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.780660 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.780989 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:43.781060 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:43.781118 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:43.781204 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 15:16:43.781251 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:16:43.781416 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:43.776467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40737 (local address 127.9.242.129:41531)
0927 15:16:43.777854 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:43.777869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:43.777966 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:43.778467 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:43.778470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:43.778478 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:43.778589 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.778594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.779424 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.779426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.780091 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:43.780099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:43.780673 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:43.780677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:43.780769 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:43.780822 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:43.781133 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:43.781140 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:16:43.781197 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:16:43.781276 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":92,"threads_started":1}
I20260927 15:16:43.807307 11915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:43.807502 11915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:43.810281 11915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Sending DeleteTablet for 3 replicas of tablet 46109cd8ec7b440990cb65d8eae81b1b
I20260927 15:16:43.811076 12194 tablet_service.cc:1514] Processing DeleteTablet for tablet 46109cd8ec7b440990cb65d8eae81b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:43 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:48836
I20260927 15:16:43.811192 12054 tablet_service.cc:1514] Processing DeleteTablet for tablet 46109cd8ec7b440990cb65d8eae81b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:43 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:34152
I20260927 15:16:43.811537 12474 tablet_replica.cc:330] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: stopping tablet replica
I20260927 15:16:43.811837 12474 raft_consensus.cc:2229] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:43.812179 12474 raft_consensus.cc:2258] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:43.812592 12334 tablet_service.cc:1514] Processing DeleteTablet for tablet 46109cd8ec7b440990cb65d8eae81b1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:43 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:44678
I20260927 15:16:43.813201 11915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:16:43.813438 12474 ts_tablet_manager.cc:1905] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:16:43.813585 11915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:43.815316 12475 tablet_replica.cc:330] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: stopping tablet replica
I20260927 15:16:43.815662 12475 raft_consensus.cc:2229] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:43.818523 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 6f61d771af8449838927f9f57c965fb5 (DEFAULT_TABLE table=test-table [id=0ff10a9d72bc4a11871d9c3c9f848404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:43.818626 12476 tablet_replica.cc:330] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: stopping tablet replica
I20260927 15:16:43.818775 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f61d771af8449838927f9f57c965fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:43.818938 12476 raft_consensus.cc:2229] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:43.819131 12476 raft_consensus.cc:2258] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:43.819484 12474 ts_tablet_manager.cc:1918] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:43.819568 12474 log.cc:1196] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/46109cd8ec7b440990cb65d8eae81b1b
I20260927 15:16:43.819880 12474 ts_tablet_manager.cc:1939] T 46109cd8ec7b440990cb65d8eae81b1b P a2fe77618ea243a8a25e965fe097f225: Deleting consensus metadata
I20260927 15:16:43.821493 12423 tablet_bootstrap.cc:492] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: Bootstrap starting.
I20260927 15:16:43.821818 12194 tablet_service.cc:1467] Processing CreateTablet for tablet 6f61d771af8449838927f9f57c965fb5 (DEFAULT_TABLE table=test-table [id=0ff10a9d72bc4a11871d9c3c9f848404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:43.821866 12475 raft_consensus.cc:2258] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:43.822067 12194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f61d771af8449838927f9f57c965fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:43.822359 12423 tablet_bootstrap.cc:654] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:43.822659 12334 tablet_service.cc:1467] Processing CreateTablet for tablet 6f61d771af8449838927f9f57c965fb5 (DEFAULT_TABLE table=test-table [id=0ff10a9d72bc4a11871d9c3c9f848404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:43.822894 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f61d771af8449838927f9f57c965fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:43.823378 12423 tablet_bootstrap.cc:492] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: No bootstrap required, opened a new log
I20260927 15:16:43.823446 12423 ts_tablet_manager.cc:1397] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:43.823828 12423 raft_consensus.cc:348] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.823938 12423 raft_consensus.cc:374] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:43.823966 12423 raft_consensus.cc:729] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7353980156374f84b85242e2c2934631, State: Initialized, Role: FOLLOWER
I20260927 15:16:43.824095 12423 consensus_queue.cc:260] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.824352 12423 ts_tablet_manager.cc:1428] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:43.824566 12424 tablet_bootstrap.cc:492] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: Bootstrap starting.
I20260927 15:16:43.824962 12476 ts_tablet_manager.cc:1905] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:43.825443 12424 tablet_bootstrap.cc:654] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:43.826318 11903 catalog_manager.cc:4841] TS a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737): tablet 46109cd8ec7b440990cb65d8eae81b1b (table test-table [id=cd9955d313c2484b828e269a2cc074db]) successfully deleted
I20260927 15:16:43.826520 12427 tablet_bootstrap.cc:492] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: Bootstrap starting.
I20260927 15:16:43.827050 12475 ts_tablet_manager.cc:1905] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:43.827443 12427 tablet_bootstrap.cc:654] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:43.828452 12476 ts_tablet_manager.cc:1918] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:43.828545 12476 log.cc:1196] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/46109cd8ec7b440990cb65d8eae81b1b
I20260927 15:16:43.828877 12476 ts_tablet_manager.cc:1939] T 46109cd8ec7b440990cb65d8eae81b1b P 233a42c709544ce19b0a3accada2e51b: Deleting consensus metadata
I20260927 15:16:43.830259 11904 catalog_manager.cc:4841] TS 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715): tablet 46109cd8ec7b440990cb65d8eae81b1b (table test-table [id=cd9955d313c2484b828e269a2cc074db]) successfully deleted
I20260927 15:16:43.830515 12475 ts_tablet_manager.cc:1918] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:43.830608 12475 log.cc:1196] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/46109cd8ec7b440990cb65d8eae81b1b
I20260927 15:16:43.830847 12475 ts_tablet_manager.cc:1939] T 46109cd8ec7b440990cb65d8eae81b1b P 7353980156374f84b85242e2c2934631: Deleting consensus metadata
I20260927 15:16:43.830897 12424 tablet_bootstrap.cc:492] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: No bootstrap required, opened a new log
I20260927 15:16:43.830987 12424 ts_tablet_manager.cc:1397] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 15:16:43.831427 12424 raft_consensus.cc:348] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.831547 12424 raft_consensus.cc:374] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:43.831578 12424 raft_consensus.cc:729] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fe77618ea243a8a25e965fe097f225, State: Initialized, Role: FOLLOWER
I20260927 15:16:43.831713 12424 consensus_queue.cc:260] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.831863 11903 catalog_manager.cc:4841] TS 7353980156374f84b85242e2c2934631 (127.9.242.129:41503): tablet 46109cd8ec7b440990cb65d8eae81b1b (table test-table [id=cd9955d313c2484b828e269a2cc074db]) successfully deleted
I20260927 15:16:43.831956 12424 ts_tablet_manager.cc:1428] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:16:43.832772 12427 tablet_bootstrap.cc:492] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: No bootstrap required, opened a new log
I20260927 15:16:43.832850 12427 ts_tablet_manager.cc:1397] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 15:16:43.833233 12427 raft_consensus.cc:348] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.833348 12427 raft_consensus.cc:374] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:43.833382 12427 raft_consensus.cc:729] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233a42c709544ce19b0a3accada2e51b, State: Initialized, Role: FOLLOWER
I20260927 15:16:43.833503 12427 consensus_queue.cc:260] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.833782 12427 ts_tablet_manager.cc:1428] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:43.978816 12469 raft_consensus.cc:482] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:43.978976 12469 raft_consensus.cc:504] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.979460 12469 leader_election.cc:288] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737), 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715)
I20260927 15:16:43.979864 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f61d771af8449838927f9f57c965fb5" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225" is_pre_election: true
I20260927 15:16:43.979923 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f61d771af8449838927f9f57c965fb5" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233a42c709544ce19b0a3accada2e51b" is_pre_election: true
I20260927 15:16:43.980000 12214 raft_consensus.cc:2454] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 0.
I20260927 15:16:43.980082 12354 raft_consensus.cc:2454] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 0.
I20260927 15:16:43.980309 12007 leader_election.cc:302] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7353980156374f84b85242e2c2934631, a2fe77618ea243a8a25e965fe097f225; no voters:
I20260927 15:16:43.980486 12469 raft_consensus.cc:2781] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:43.980538 12469 raft_consensus.cc:482] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:43.980576 12469 raft_consensus.cc:3037] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.981864 12469 raft_consensus.cc:504] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.982218 12469 leader_election.cc:288] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 election: Requested vote from peers a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737), 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715)
I20260927 15:16:43.982553 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f61d771af8449838927f9f57c965fb5" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225"
I20260927 15:16:43.982561 12354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f61d771af8449838927f9f57c965fb5" candidate_uuid: "7353980156374f84b85242e2c2934631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233a42c709544ce19b0a3accada2e51b"
I20260927 15:16:43.982690 12354 raft_consensus.cc:3037] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.982698 12214 raft_consensus.cc:3037] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:43.983816 12214 raft_consensus.cc:2454] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 1.
I20260927 15:16:43.983826 12354 raft_consensus.cc:2454] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7353980156374f84b85242e2c2934631 in term 1.
I20260927 15:16:43.984126 12007 leader_election.cc:302] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7353980156374f84b85242e2c2934631, a2fe77618ea243a8a25e965fe097f225; no voters:
I20260927 15:16:43.984301 12469 raft_consensus.cc:2781] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:43.984385 12469 raft_consensus.cc:686] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 1 LEADER]: Becoming Leader. State: Replica: 7353980156374f84b85242e2c2934631, State: Running, Role: LEADER
I20260927 15:16:43.984520 12469 consensus_queue.cc:237] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:43.986224 11915 catalog_manager.cc:5495] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7353980156374f84b85242e2c2934631 (127.9.242.129). New cstate: current_term: 1 leader_uuid: "7353980156374f84b85242e2c2934631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } health_report { overall_health: UNKNOWN } } }
I20260927 15:16:44.029516 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:44.026652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.9.242.129:41503)
0927 15:16:44.026751 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:44.026754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:44.026767 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:16:44.026879 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:44.026882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:44.026921 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:44.026979 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:44.027153 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:44.027736 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:44.028562 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:44.028773 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:44.028855 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:44.029083 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:44.029150 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:44.029209 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:44.029327 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 15:16:44.029381 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 15:16:44.037724 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:44.034991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:44843 (local address 127.9.242.129:41503)
0927 15:16:44.035077 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:44.035081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:44.035094 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:16:44.035216 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:44.035219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:44.035254 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:44.035307 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:44.035522 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:44.036020 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:44.036801 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:44.036994 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:44.037071 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:44.037298 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:44.037357 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:44.037410 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:44.037491 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 15:16:44.037528 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":22}
I20260927 15:16:44.037724 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:44.034891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.9.242.129:44843)
0927 15:16:44.035028 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:44.035044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:44.035122 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:44.035327 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:44.035330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:44.035336 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:44.035436 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:44.035441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:44.036112 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:44.036122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:44.036697 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:44.036705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:44.037082 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:44.037085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:44.037177 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:44.037202 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:44.037425 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:44.037430 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:16:44.037484 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 15:16:44.037559 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":18}
I20260927 15:16:44.042626 12354 raft_consensus.cc:1261] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Refusing update from remote peer 7353980156374f84b85242e2c2934631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:44.042917 12214 raft_consensus.cc:1261] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Refusing update from remote peer 7353980156374f84b85242e2c2934631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:44.042963 12469 consensus_queue.cc:1035] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:44.043226 12428 consensus_queue.cc:1035] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:16:44.055094 12453 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:16:45.606087 11916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:45.606263 11916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:45.608589 11916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Sending DeleteTablet for 3 replicas of tablet 6f61d771af8449838927f9f57c965fb5
I20260927 15:16:45.609076 12053 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f61d771af8449838927f9f57c965fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:45 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:34152
I20260927 15:16:45.609139 12334 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f61d771af8449838927f9f57c965fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:45 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:44678
I20260927 15:16:45.609136 12194 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f61d771af8449838927f9f57c965fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:45 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:48836
I20260927 15:16:45.609467 12506 tablet_replica.cc:330] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: stopping tablet replica
I20260927 15:16:45.609476 12505 tablet_replica.cc:330] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: stopping tablet replica
I20260927 15:16:45.609593 12505 raft_consensus.cc:2229] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:45.609638 12506 raft_consensus.cc:2229] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:45.609733 12505 raft_consensus.cc:2258] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:45.609697 12504 tablet_replica.cc:330] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: stopping tablet replica
I20260927 15:16:45.609825 12506 raft_consensus.cc:2258] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:45.609985 12504 raft_consensus.cc:2229] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:45.610386 12504 raft_consensus.cc:2258] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:45.610510 12505 ts_tablet_manager.cc:1905] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:45.611076 12506 ts_tablet_manager.cc:1905] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:45.611544 12504 ts_tablet_manager.cc:1905] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:45.613250 12505 ts_tablet_manager.cc:1918] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:16:45.613322 12505 log.cc:1196] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/6f61d771af8449838927f9f57c965fb5
I20260927 15:16:45.613585 12505 ts_tablet_manager.cc:1939] T 6f61d771af8449838927f9f57c965fb5 P 233a42c709544ce19b0a3accada2e51b: Deleting consensus metadata
I20260927 15:16:45.613953 12506 ts_tablet_manager.cc:1918] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:16:45.614017 12506 log.cc:1196] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/6f61d771af8449838927f9f57c965fb5
I20260927 15:16:45.614215 12506 ts_tablet_manager.cc:1939] T 6f61d771af8449838927f9f57c965fb5 P a2fe77618ea243a8a25e965fe097f225: Deleting consensus metadata
I20260927 15:16:45.614455 11904 catalog_manager.cc:4841] TS 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715): tablet 6f61d771af8449838927f9f57c965fb5 (table test-table [id=0ff10a9d72bc4a11871d9c3c9f848404]) successfully deleted
I20260927 15:16:45.615195 11903 catalog_manager.cc:4841] TS a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737): tablet 6f61d771af8449838927f9f57c965fb5 (table test-table [id=0ff10a9d72bc4a11871d9c3c9f848404]) successfully deleted
I20260927 15:16:45.617439 12504 ts_tablet_manager.cc:1918] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:16:45.617503 12504 log.cc:1196] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/6f61d771af8449838927f9f57c965fb5
I20260927 15:16:45.617779 12504 ts_tablet_manager.cc:1939] T 6f61d771af8449838927f9f57c965fb5 P 7353980156374f84b85242e2c2934631: Deleting consensus metadata
I20260927 15:16:45.618454 11903 catalog_manager.cc:4841] TS 7353980156374f84b85242e2c2934631 (127.9.242.129:41503): tablet 6f61d771af8449838927f9f57c965fb5 (table test-table [id=0ff10a9d72bc4a11871d9c3c9f848404]) successfully deleted
Sep 27 15:16:45 dist-test-slave-m4lt krb5kdc[11872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:16:45 dist-test-slave-m4lt krb5kdc[11872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:45.642745 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.627718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.9.242.190:42157)
0927 15:16:45.628000 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.628007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.628037 (+ 30us) server_negotiation.cc:407] Connection header received
0927 15:16:45.628076 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.628079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.628121 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.628187 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:45.628969 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.629841 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.630718 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.630952 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.631033 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.633503 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:45.633506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:45.633508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:45.633534 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:45.640932 (+ 7398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.641445 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.641448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.641450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.641538 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:45.641869 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:45.641872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:45.641874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:45.642073 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:45.642198 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.642445 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.642571 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260927 15:16:45.645612 11916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:45.646145 11916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:45.650767 12194 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 (DEFAULT_TABLE table=test-table [id=1dcf93153f8345cba61289ac7d11c2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:45.650796 12053 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 (DEFAULT_TABLE table=test-table [id=1dcf93153f8345cba61289ac7d11c2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:45.650906 12334 tablet_service.cc:1467] Processing CreateTablet for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 (DEFAULT_TABLE table=test-table [id=1dcf93153f8345cba61289ac7d11c2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:45.651060 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:45.651060 12194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:45.651161 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:45.653970 12515 tablet_bootstrap.cc:492] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: Bootstrap starting.
I20260927 15:16:45.654129 12516 tablet_bootstrap.cc:492] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: Bootstrap starting.
I20260927 15:16:45.654915 12516 tablet_bootstrap.cc:654] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:45.654989 12515 tablet_bootstrap.cc:654] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:45.655545 12517 tablet_bootstrap.cc:492] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: Bootstrap starting.
I20260927 15:16:45.655989 12516 tablet_bootstrap.cc:492] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: No bootstrap required, opened a new log
I20260927 15:16:45.656080 12516 ts_tablet_manager.cc:1397] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:45.656332 12517 tablet_bootstrap.cc:654] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:45.656540 12515 tablet_bootstrap.cc:492] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: No bootstrap required, opened a new log
I20260927 15:16:45.656512 12516 raft_consensus.cc:348] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.656618 12516 raft_consensus.cc:374] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:45.656613 12515 ts_tablet_manager.cc:1397] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:45.656649 12516 raft_consensus.cc:729] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7353980156374f84b85242e2c2934631, State: Initialized, Role: FOLLOWER
I20260927 15:16:45.656808 12516 consensus_queue.cc:260] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.657086 12516 ts_tablet_manager.cc:1428] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:45.657325 12517 tablet_bootstrap.cc:492] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: No bootstrap required, opened a new log
I20260927 15:16:45.657408 12517 ts_tablet_manager.cc:1397] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:45.657485 12515 raft_consensus.cc:348] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.657588 12515 raft_consensus.cc:374] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:45.657634 12515 raft_consensus.cc:729] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233a42c709544ce19b0a3accada2e51b, State: Initialized, Role: FOLLOWER
I20260927 15:16:45.657778 12515 consensus_queue.cc:260] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.657802 12517 raft_consensus.cc:348] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.657886 12517 raft_consensus.cc:374] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:45.657919 12517 raft_consensus.cc:729] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fe77618ea243a8a25e965fe097f225, State: Initialized, Role: FOLLOWER
I20260927 15:16:45.658035 12517 consensus_queue.cc:260] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.658201 12515 ts_tablet_manager.cc:1428] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:45.658375 12517 ts_tablet_manager.cc:1428] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:45.762395 12521 raft_consensus.cc:482] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:45.762537 12521 raft_consensus.cc:504] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.763547 12521 leader_election.cc:288] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7353980156374f84b85242e2c2934631 (127.9.242.129:41503), a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737)
I20260927 15:16:45.767684 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.763905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:39577 (local address 127.9.242.129:41503)
0927 15:16:45.764178 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.764186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.764202 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:16:45.764384 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.764387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.764442 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.764521 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:45.764936 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.765504 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.766711 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.767117 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.767221 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.767284 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.767484 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.767554 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260927 15:16:45.767684 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.764347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:37939 (local address 127.9.242.130:40737)
0927 15:16:45.764631 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.764648 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.764660 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:16:45.764705 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.764708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.764750 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.764833 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:45.765069 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.765604 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.766711 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.767124 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.767233 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.767295 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.767469 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.767542 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260927 15:16:45.767695 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.764204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40737 (local address 127.9.242.131:37939)
0927 15:16:45.764474 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:45.764487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:45.764585 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:45.764841 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:45.764843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:45.764856 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:45.764972 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.764978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.765757 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.765760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.766582 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.766591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.767240 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.767243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.767355 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.767375 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:16:45.767436 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 15:16:45.767542 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":45,"thread_start_us":115,"threads_started":1}
I20260927 15:16:45.767700 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.763805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.9.242.131:39577)
0927 15:16:45.764104 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:45.764118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:45.764221 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:45.764639 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:45.764643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:45.764662 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:45.764810 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.764817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.765609 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.765613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.766575 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.766585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.767233 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.767237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.767364 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.767386 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:16:45.767452 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:16:45.767552 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":81,"thread_start_us":109,"threads_started":1}
I20260927 15:16:45.768105 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7ec5a0add04bfbb4881fc2c5ab3315" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225" is_pre_election: true
I20260927 15:16:45.768258 12214 raft_consensus.cc:2454] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 0.
I20260927 15:16:45.768523 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7ec5a0add04bfbb4881fc2c5ab3315" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7353980156374f84b85242e2c2934631" is_pre_election: true
I20260927 15:16:45.768577 12287 leader_election.cc:302] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233a42c709544ce19b0a3accada2e51b, a2fe77618ea243a8a25e965fe097f225; no voters:
I20260927 15:16:45.768636 12074 raft_consensus.cc:2454] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 0.
I20260927 15:16:45.768834 12521 raft_consensus.cc:2781] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:45.768882 12521 raft_consensus.cc:482] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:45.768915 12521 raft_consensus.cc:3037] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:45.769851 12521 raft_consensus.cc:504] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.770143 12521 leader_election.cc:288] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 election: Requested vote from peers 7353980156374f84b85242e2c2934631 (127.9.242.129:41503), a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737)
I20260927 15:16:45.770473 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7ec5a0add04bfbb4881fc2c5ab3315" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7353980156374f84b85242e2c2934631"
I20260927 15:16:45.770520 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f7ec5a0add04bfbb4881fc2c5ab3315" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225"
I20260927 15:16:45.770569 12074 raft_consensus.cc:3037] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:45.770638 12214 raft_consensus.cc:3037] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:45.771400 12074 raft_consensus.cc:2454] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 1.
I20260927 15:16:45.771533 12214 raft_consensus.cc:2454] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 1.
I20260927 15:16:45.771672 12287 leader_election.cc:302] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233a42c709544ce19b0a3accada2e51b, 7353980156374f84b85242e2c2934631; no voters:
I20260927 15:16:45.771828 12521 raft_consensus.cc:2781] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:45.772032 12521 raft_consensus.cc:686] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 1 LEADER]: Becoming Leader. State: Replica: 233a42c709544ce19b0a3accada2e51b, State: Running, Role: LEADER
I20260927 15:16:45.772267 12521 consensus_queue.cc:237] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.774616 11917 catalog_manager.cc:5495] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b reported cstate change: term changed from 0 to 1, leader changed from <none> to 233a42c709544ce19b0a3accada2e51b (127.9.242.131). New cstate: current_term: 1 leader_uuid: "233a42c709544ce19b0a3accada2e51b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } health_report { overall_health: HEALTHY } } }
I20260927 15:16:45.817124 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.813513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.9.242.131:42715)
0927 15:16:45.813769 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.813779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.813791 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:16:45.813873 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.813875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.813926 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.814027 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:45.814303 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.814855 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.815691 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.815919 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.816004 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.816279 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:45.816741 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:45.816859 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.816920 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.816971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260927 15:16:45.827219 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.824048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:48623 (local address 127.9.242.129:41503)
0927 15:16:45.824172 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.824177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.824205 (+ 28us) server_negotiation.cc:407] Connection header received
0927 15:16:45.824357 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.824360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.824405 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.824465 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:45.824698 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.825197 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.826129 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.826395 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.826484 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.826761 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:45.826833 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:45.826888 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.827020 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.827073 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:16:45.827230 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.823840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.9.242.131:48623)
0927 15:16:45.824137 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:45.824153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:45.824254 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:45.824487 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:45.824490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:45.824497 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:45.824609 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.824613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.825310 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.825314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.826024 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.826033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.826496 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.826499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.826597 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.826640 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:45.826904 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:45.826910 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:16:45.827008 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:16:45.827084 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20260927 15:16:45.832948 12074 raft_consensus.cc:1261] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Refusing update from remote peer 233a42c709544ce19b0a3accada2e51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:45.832974 12214 raft_consensus.cc:1261] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Refusing update from remote peer 233a42c709544ce19b0a3accada2e51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:45.833436 12521 consensus_queue.cc:1035] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:45.833698 12527 consensus_queue.cc:1035] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:45.849975 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.846812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:44987 (local address 127.9.242.131:42715)
0927 15:16:45.846934 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:45.846938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:45.847073 (+ 135us) server_negotiation.cc:407] Connection header received
0927 15:16:45.847224 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:45.847226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:45.847272 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:45.847349 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:45.847597 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.848146 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.848934 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.849143 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.849235 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.849518 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:45.849597 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:45.849690 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:45.849769 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 15:16:45.849819 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:16:45.850044 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:45.846732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:42715 (local address 127.9.242.129:44987)
0927 15:16:45.846992 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:45.847006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:45.847136 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:45.847369 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:45.847371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:45.847381 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:16:45.847489 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.847496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.848229 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.848231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.848838 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:45.848846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:45.849278 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:45.849282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:45.849365 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:45.849400 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:16:45.849698 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:16:45.849704 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:16:45.849760 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 15:16:45.849901 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":113,"threads_started":1}
I20260927 15:16:45.872628 11916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:45.872788 11916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:45.874799 11916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Sending DeleteTablet for 3 replicas of tablet 7f7ec5a0add04bfbb4881fc2c5ab3315
I20260927 15:16:45.875429 12194 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:45 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:48836
I20260927 15:16:45.875486 12334 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:45 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:44678
I20260927 15:16:45.875612 12506 tablet_replica.cc:330] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: stopping tablet replica
I20260927 15:16:45.875618 12505 tablet_replica.cc:330] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: stopping tablet replica
I20260927 15:16:45.875708 12505 raft_consensus.cc:2229] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:45.875710 12506 raft_consensus.cc:2229] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:45.875813 12506 raft_consensus.cc:2258] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:45.876016 12505 raft_consensus.cc:2258] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:45.876397 12053 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:45 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:34152
I20260927 15:16:45.876565 12504 tablet_replica.cc:330] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: stopping tablet replica
I20260927 15:16:45.876659 12504 raft_consensus.cc:2229] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:45.876754 12504 raft_consensus.cc:2258] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:45.876739 12505 ts_tablet_manager.cc:1905] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:45.877619 12506 ts_tablet_manager.cc:1905] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:45.879881 12505 ts_tablet_manager.cc:1918] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:45.879951 12505 log.cc:1196] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/7f7ec5a0add04bfbb4881fc2c5ab3315
I20260927 15:16:45.880055 12506 ts_tablet_manager.cc:1918] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:45.880111 12506 log.cc:1196] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/7f7ec5a0add04bfbb4881fc2c5ab3315
I20260927 15:16:45.880179 12505 ts_tablet_manager.cc:1939] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 233a42c709544ce19b0a3accada2e51b: Deleting consensus metadata
I20260927 15:16:45.880324 12506 ts_tablet_manager.cc:1939] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P a2fe77618ea243a8a25e965fe097f225: Deleting consensus metadata
I20260927 15:16:45.880923 11904 catalog_manager.cc:4841] TS 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715): tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 (table test-table [id=1dcf93153f8345cba61289ac7d11c2b7]) successfully deleted
I20260927 15:16:45.881345 12504 ts_tablet_manager.cc:1905] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:45.881701 11916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:45.882157 11916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:45.883843 12504 ts_tablet_manager.cc:1918] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:16:45.883908 12504 log.cc:1196] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/7f7ec5a0add04bfbb4881fc2c5ab3315
I20260927 15:16:45.884153 12504 ts_tablet_manager.cc:1939] T 7f7ec5a0add04bfbb4881fc2c5ab3315 P 7353980156374f84b85242e2c2934631: Deleting consensus metadata
I20260927 15:16:45.884891 11903 catalog_manager.cc:4841] TS a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737): tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 (table test-table [id=1dcf93153f8345cba61289ac7d11c2b7]) successfully deleted
I20260927 15:16:45.885015 11903 catalog_manager.cc:4841] TS 7353980156374f84b85242e2c2934631 (127.9.242.129:41503): tablet 7f7ec5a0add04bfbb4881fc2c5ab3315 (table test-table [id=1dcf93153f8345cba61289ac7d11c2b7]) successfully deleted
I20260927 15:16:45.887164 12194 tablet_service.cc:1467] Processing CreateTablet for tablet a791b0ea7b034a47bbee1b4786909433 (DEFAULT_TABLE table=test-table [id=335ecb12670f4b2ab5e9e5be5c784320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:45.887437 12194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a791b0ea7b034a47bbee1b4786909433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:45.887408 12334 tablet_service.cc:1467] Processing CreateTablet for tablet a791b0ea7b034a47bbee1b4786909433 (DEFAULT_TABLE table=test-table [id=335ecb12670f4b2ab5e9e5be5c784320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:45.887616 12334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a791b0ea7b034a47bbee1b4786909433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:45.889643 12517 tablet_bootstrap.cc:492] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225: Bootstrap starting.
I20260927 15:16:45.889745 12515 tablet_bootstrap.cc:492] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: Bootstrap starting.
I20260927 15:16:45.890398 12053 tablet_service.cc:1467] Processing CreateTablet for tablet a791b0ea7b034a47bbee1b4786909433 (DEFAULT_TABLE table=test-table [id=335ecb12670f4b2ab5e9e5be5c784320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:45.890482 12517 tablet_bootstrap.cc:654] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:45.890635 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a791b0ea7b034a47bbee1b4786909433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:45.890635 12515 tablet_bootstrap.cc:654] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:45.892445 12515 tablet_bootstrap.cc:492] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: No bootstrap required, opened a new log
I20260927 15:16:45.892521 12515 ts_tablet_manager.cc:1397] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 15:16:45.892920 12516 tablet_bootstrap.cc:492] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631: Bootstrap starting.
I20260927 15:16:45.892987 12515 raft_consensus.cc:348] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.893082 12515 raft_consensus.cc:374] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:45.893132 12515 raft_consensus.cc:729] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233a42c709544ce19b0a3accada2e51b, State: Initialized, Role: FOLLOWER
I20260927 15:16:45.893244 12515 consensus_queue.cc:260] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.893486 12515 ts_tablet_manager.cc:1428] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:16:45.893703 12516 tablet_bootstrap.cc:654] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:45.894686 12516 tablet_bootstrap.cc:492] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631: No bootstrap required, opened a new log
I20260927 15:16:45.894779 12516 ts_tablet_manager.cc:1397] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 15:16:45.895150 12516 raft_consensus.cc:348] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.895256 12516 raft_consensus.cc:374] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:45.895284 12516 raft_consensus.cc:729] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7353980156374f84b85242e2c2934631, State: Initialized, Role: FOLLOWER
I20260927 15:16:45.895390 12516 consensus_queue.cc:260] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.895618 12516 ts_tablet_manager.cc:1428] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:16:45.896169 12517 tablet_bootstrap.cc:492] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225: No bootstrap required, opened a new log
I20260927 15:16:45.896263 12517 ts_tablet_manager.cc:1397] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 15:16:45.896624 12517 raft_consensus.cc:348] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.896701 12517 raft_consensus.cc:374] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:45.896719 12517 raft_consensus.cc:729] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fe77618ea243a8a25e965fe097f225, State: Initialized, Role: FOLLOWER
I20260927 15:16:45.896821 12517 consensus_queue.cc:260] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:45.897066 12517 ts_tablet_manager.cc:1428] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:16:46.001581 12521 raft_consensus.cc:482] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:46.001772 12521 raft_consensus.cc:504] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:46.002197 12521 leader_election.cc:288] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7353980156374f84b85242e2c2934631 (127.9.242.129:41503), a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737)
I20260927 15:16:46.002652 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a791b0ea7b034a47bbee1b4786909433" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7353980156374f84b85242e2c2934631" is_pre_election: true
I20260927 15:16:46.002740 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a791b0ea7b034a47bbee1b4786909433" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225" is_pre_election: true
I20260927 15:16:46.002817 12074 raft_consensus.cc:2454] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 0.
I20260927 15:16:46.002874 12214 raft_consensus.cc:2454] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 0.
I20260927 15:16:46.003121 12287 leader_election.cc:302] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233a42c709544ce19b0a3accada2e51b, 7353980156374f84b85242e2c2934631; no voters:
I20260927 15:16:46.003333 12521 raft_consensus.cc:2781] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:46.003396 12521 raft_consensus.cc:482] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:46.003434 12521 raft_consensus.cc:3037] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:46.004415 12521 raft_consensus.cc:504] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:46.004753 12521 leader_election.cc:288] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 election: Requested vote from peers 7353980156374f84b85242e2c2934631 (127.9.242.129:41503), a2fe77618ea243a8a25e965fe097f225 (127.9.242.130:40737)
I20260927 15:16:46.005116 12074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a791b0ea7b034a47bbee1b4786909433" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7353980156374f84b85242e2c2934631"
I20260927 15:16:46.005162 12214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a791b0ea7b034a47bbee1b4786909433" candidate_uuid: "233a42c709544ce19b0a3accada2e51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fe77618ea243a8a25e965fe097f225"
I20260927 15:16:46.005249 12074 raft_consensus.cc:3037] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:46.005264 12214 raft_consensus.cc:3037] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:46.006413 12074 raft_consensus.cc:2454] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 1.
I20260927 15:16:46.006413 12214 raft_consensus.cc:2454] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233a42c709544ce19b0a3accada2e51b in term 1.
I20260927 15:16:46.006714 12287 leader_election.cc:302] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233a42c709544ce19b0a3accada2e51b, 7353980156374f84b85242e2c2934631; no voters:
I20260927 15:16:46.007001 12521 raft_consensus.cc:2781] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:46.007077 12521 raft_consensus.cc:686] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 1 LEADER]: Becoming Leader. State: Replica: 233a42c709544ce19b0a3accada2e51b, State: Running, Role: LEADER
I20260927 15:16:46.007212 12521 consensus_queue.cc:237] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } }
I20260927 15:16:46.008903 11916 catalog_manager.cc:5495] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b reported cstate change: term changed from 0 to 1, leader changed from <none> to 233a42c709544ce19b0a3accada2e51b (127.9.242.131). New cstate: current_term: 1 leader_uuid: "233a42c709544ce19b0a3accada2e51b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233a42c709544ce19b0a3accada2e51b" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 42715 } health_report { overall_health: HEALTHY } } }
I20260927 15:16:46.029387 12214 raft_consensus.cc:1261] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Refusing update from remote peer 233a42c709544ce19b0a3accada2e51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:46.029448 12074 raft_consensus.cc:1261] T a791b0ea7b034a47bbee1b4786909433 P 7353980156374f84b85242e2c2934631 [term 1 FOLLOWER]: Refusing update from remote peer 233a42c709544ce19b0a3accada2e51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:46.029800 12521 consensus_queue.cc:1035] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2fe77618ea243a8a25e965fe097f225" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:46.029968 12536 consensus_queue.cc:1035] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7353980156374f84b85242e2c2934631" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:16:46.041124 12489 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 15:16:47.588857 11917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:47.589015 11917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:47.591315 11917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67f79503804e426c92f6182039de2777: Sending DeleteTablet for 3 replicas of tablet a791b0ea7b034a47bbee1b4786909433
I20260927 15:16:47.591809 12053 tablet_service.cc:1514] Processing DeleteTablet for tablet a791b0ea7b034a47bbee1b4786909433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:47 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:34152
I20260927 15:16:47.591910 12334 tablet_service.cc:1514] Processing DeleteTablet for tablet a791b0ea7b034a47bbee1b4786909433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:47 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:44678
I20260927 15:16:47.591938 12194 tablet_service.cc:1514] Processing DeleteTablet for tablet a791b0ea7b034a47bbee1b4786909433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:47 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:48836
I20260927 15:16:47.592193 12569 tablet_replica.cc:330] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: stopping tablet replica
I20260927 15:16:47.592242 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 11989
I20260927 15:16:47.592974 12569 raft_consensus.cc:2229] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:47.593148 12569 raft_consensus.cc:2258] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:47.593698 12569 ts_tablet_manager.cc:1905] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:47.595763 12569 ts_tablet_manager.cc:1918] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 15:16:47.595896 12569 log.cc:1196] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/a791b0ea7b034a47bbee1b4786909433
I20260927 15:16:47.596177 12569 ts_tablet_manager.cc:1939] T a791b0ea7b034a47bbee1b4786909433 P 233a42c709544ce19b0a3accada2e51b: Deleting consensus metadata
I20260927 15:16:47.596730 12570 tablet_replica.cc:330] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225: stopping tablet replica
I20260927 15:16:47.596797 11904 catalog_manager.cc:4841] TS 233a42c709544ce19b0a3accada2e51b (127.9.242.131:42715): tablet a791b0ea7b034a47bbee1b4786909433 (table test-table [id=335ecb12670f4b2ab5e9e5be5c784320]) successfully deleted
I20260927 15:16:47.596863 12570 raft_consensus.cc:2229] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:47.596995 12570 raft_consensus.cc:2258] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:47.597510 12570 ts_tablet_manager.cc:1905] T a791b0ea7b034a47bbee1b4786909433 P a2fe77618ea243a8a25e965fe097f225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:47.598459 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 12129
W20260927 15:16:47.599613 11903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:41503 (error 108)
I20260927 15:16:47.600754 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:47.600360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:41503 (local address 127.0.0.1:34666)
0927 15:16:47.600598 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:47.600670 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
W20260927 15:16:47.600876 11903 catalog_manager.cc:4584] TS 7353980156374f84b85242e2c2934631 (127.9.242.129:41503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a791b0ea7b034a47bbee1b4786909433: Network error: Client connection negotiation failed: client connection to 127.9.242.129:41503: connect: Connection refused (error 111)
I20260927 15:16:47.603804 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 12269
I20260927 15:16:47.604768 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:47.604518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40737 (local address 127.0.0.1:51932)
0927 15:16:47.604637 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:47.604689 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.130:40737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:16:47.609247 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 11885
2026-09-27T15:16:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6694 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:47 dist-test-slave-m4lt krb5kdc[12575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:47 dist-test-slave-m4lt krb5kdc[12575](info): set up 2 sockets
Sep 27 15:16:47 dist-test-slave-m4lt krb5kdc[12575](info): commencing operation
krb5kdc: starting...
W20260927 15:16:48.569350 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:48 dist-test-slave-m4lt krb5kdc[12575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:48.685376 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39803
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:39803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:48.791360 12588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:48.791643 12588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:48.791709 12588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:48.795805 12588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:48.795876 12588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:48.795902 12588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:48.795924 12588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:48.795944 12588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:48.801249 12588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:39803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:48.802314 12588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:48.803316 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:48.812247 12594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:48.812333 12593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:48.812415 12588 server_base.cc:1034] running on GCE node
W20260927 15:16:48.812333 12596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:48.813037 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:48.814188 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:48.815356 12588 hybrid_clock.cc:648] HybridClock initialized: now 1790522208815343 us; error 26 us; skew 500 ppm
Sep 27 15:16:48 dist-test-slave-m4lt krb5kdc[12575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522208, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:48.818082 12588 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:48.819042 12588 webserver.cc:460] Webserver started at http://127.9.242.190:44951/ using document root <none> and password file <none>
I20260927 15:16:48.819584 12588 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:48.819649 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:48.819839 12588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:48.821970 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "fd22a65c4a8a4ffa958a8ce824624e33"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-m4lt"
server_key: "48afecc44c13cee7729b2887b7ce26b1"
server_key_iv: "ca4833e8c9c57757b1303143748d5ed2"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.822508 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "fd22a65c4a8a4ffa958a8ce824624e33"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-m4lt"
server_key: "48afecc44c13cee7729b2887b7ce26b1"
server_key_iv: "ca4833e8c9c57757b1303143748d5ed2"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.826334 12588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:16:48.828970 12603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:48.830405 12588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:48.830533 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "fd22a65c4a8a4ffa958a8ce824624e33"
format_stamp: "Formatted at 2026-09-27 15:16:48 on dist-test-slave-m4lt"
server_key: "48afecc44c13cee7729b2887b7ce26b1"
server_key_iv: "ca4833e8c9c57757b1303143748d5ed2"
server_key_version: "encryptionkey@0"
I20260927 15:16:48.830657 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:48.842020 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:48.845240 12588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:48.845440 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:48.852578 12588 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:39803
I20260927 15:16:48.852602 12655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:39803 every 8 connection(s)
I20260927 15:16:48.853488 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:48.856226 12656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:48.861055 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 12588
I20260927 15:16:48.861162 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:48.861415 10186 external_mini_cluster.cc:1428] Setting key 6285c6ee6639e4cd58b102ad9de40c9b
I20260927 15:16:48.861742 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33: Bootstrap starting.
I20260927 15:16:48.864126 12656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:48.864998 12656 log.cc:824] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:48.867187 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33: No bootstrap required, opened a new log
Sep 27 15:16:48 dist-test-slave-m4lt krb5kdc[12575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:48.871403 12656 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39803 } }
I20260927 15:16:48.871677 12656 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:48.871738 12656 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd22a65c4a8a4ffa958a8ce824624e33, State: Initialized, Role: FOLLOWER
I20260927 15:16:48.872244 12656 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39803 } }
I20260927 15:16:48.872406 12656 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:48.872475 12656 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:48.872579 12656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:48.873587 12656 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39803 } }
I20260927 15:16:48.873970 12656 leader_election.cc:302] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd22a65c4a8a4ffa958a8ce824624e33; no voters:
I20260927 15:16:48.874451 12656 leader_election.cc:288] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:48.874594 12661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:48.874836 12661 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [term 1 LEADER]: Becoming Leader. State: Replica: fd22a65c4a8a4ffa958a8ce824624e33, State: Running, Role: LEADER
I20260927 15:16:48.875185 12661 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39803 } }
I20260927 15:16:48.875875 12656 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:48.876358 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:48.862834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.9.242.190:39803)
0927 15:16:48.863237 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:48.863252 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:48.863287 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:16:48.863906 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:48.863922 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:48.864335 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:48.864627 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:48.865499 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.866672 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.867862 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:48.868325 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:48.868446 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:48.871273 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:48.871278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:48.871292 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:48.871320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:48.873042 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.873517 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.873521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.873526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.873693 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:48.873954 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:48.873956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:48.873958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:48.874530 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:48.874760 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:48.875033 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:16:48.875410 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":100,"threads_started":1}
I20260927 15:16:48.876432 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39803 } } }
I20260927 15:16:48.876518 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:48.876713 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd22a65c4a8a4ffa958a8ce824624e33. Latest consensus state: current_term: 1 leader_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd22a65c4a8a4ffa958a8ce824624e33" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39803 } } }
I20260927 15:16:48.876788 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:48.878700 12666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:48.881165 12666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:48.886582 12666 catalog_manager.cc:1347] Generated new cluster ID: 42e83d8389ac4643a969bb0e13b11816
I20260927 15:16:48.886647 12666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:48.903759 12666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:48.905053 12666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:48.915515 12666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd22a65c4a8a4ffa958a8ce824624e33: Generated new TSK 0
I20260927 15:16:48.916258 12666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:48.948668 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39803
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:49.059692 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:49.059957 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:49.060025 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:49.064134 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:49.064200 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:49.064282 12682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:49.069641 12682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:49.070749 12682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:49.071753 12682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:49.081264 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:49.081257 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:49.081292 12682 server_base.cc:1034] running on GCE node
W20260927 15:16:49.081255 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:49.081969 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:49.082631 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:49.083833 12682 hybrid_clock.cc:648] HybridClock initialized: now 1790522209083816 us; error 34 us; skew 500 ppm
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[12575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:49.086653 12682 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:49.087553 12682 webserver.cc:460] Webserver started at http://127.9.242.129:35709/ using document root <none> and password file <none>
I20260927 15:16:49.088061 12682 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:49.088115 12682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:49.088286 12682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:49.090309 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "9ea0ceacce05409a85e7345e67d73af4"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "fd6013280440fccd4f35fe681ff9fbf0"
server_key_iv: "f8fd35d00615b0bd2a67b9d044b7d6c0"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.090808 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "9ea0ceacce05409a85e7345e67d73af4"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "fd6013280440fccd4f35fe681ff9fbf0"
server_key_iv: "f8fd35d00615b0bd2a67b9d044b7d6c0"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.094421 12682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:49.096838 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.098188 12682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:49.098289 12682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "9ea0ceacce05409a85e7345e67d73af4"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "fd6013280440fccd4f35fe681ff9fbf0"
server_key_iv: "f8fd35d00615b0bd2a67b9d044b7d6c0"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.098402 12682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:49.133000 12682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:49.136006 12682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:49.136171 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:49.136736 12682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:49.137526 12682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:49.137568 12682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.137609 12682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:49.137637 12682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.145939 12682 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:43863
I20260927 15:16:49.146034 12810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:43863 every 8 connection(s)
I20260927 15:16:49.146857 12682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:49.155328 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 12682
I20260927 15:16:49.155423 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:49.155666 10186 external_mini_cluster.cc:1428] Setting key d74a39022e6ad6e7651fd44235d3d1da
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[12575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:49.159920 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.148549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:37717 (local address 127.9.242.190:39803)
0927 15:16:49.148720 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.148723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.149749 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 15:16:49.150623 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.150626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.150681 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.150754 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:49.151883 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.152443 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.153550 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.153827 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.153915 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.156362 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:49.156366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:49.156369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:49.156396 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:49.157889 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.158335 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.158338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.158340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.158417 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.158831 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.158834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.158835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.159005 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:49.159087 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.159667 (+ 580us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.159801 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 15:16:49.160944 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.148964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39803 (local address 127.9.242.129:37717)
0927 15:16:49.149578 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.149621 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:49.150421 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:49.150927 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:49.150934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:49.151287 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:49.151746 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.151759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.152563 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.152566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.153437 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.153446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.153961 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.153966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.154154 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.154683 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:49.154700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:49.156217 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:49.157986 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:49.157991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:49.158002 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:49.158250 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:49.158506 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:49.158509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:49.158511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:49.158686 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:49.159098 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:49.159104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:49.159421 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 15:16:49.159624 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 15:16:49.159945 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":106,"threads_started":1}
I20260927 15:16:49.162215 12811 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39803
I20260927 15:16:49.162436 12811 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:49.162878 12811 heartbeater.cc:511] Master 127.9.242.190:39803 requested a full tablet report, sending...
I20260927 15:16:49.164354 12617 ts_manager.cc:194] Registered new tserver with Master: 9ea0ceacce05409a85e7345e67d73af4 (127.9.242.129:43863)
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:16:49.165484 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:37717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:49.180727 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39803
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:49.286731 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:49.286980 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:49.287042 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:49.291088 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:49.291149 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:49.291234 12816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:49.296439 12816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:49.297443 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:49.298451 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:49.307514 12821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:49.307560 12824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:49.307561 12822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:49.307744 12816 server_base.cc:1034] running on GCE node
I20260927 15:16:49.308167 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:49.308794 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:49.309938 12816 hybrid_clock.cc:648] HybridClock initialized: now 1790522209309924 us; error 28 us; skew 500 ppm
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[12575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:49.312467 12816 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:49.313338 12816 webserver.cc:460] Webserver started at http://127.9.242.130:40419/ using document root <none> and password file <none>
I20260927 15:16:49.313894 12816 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:49.313958 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:49.314150 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:49.316090 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "65524d2710434bcabffddc1d8e3ed215"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "c2b272e5591b981c636e7b5f7aca37b5"
server_key_iv: "f1a25a640c440ad384c53bce32c51d60"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.316614 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "65524d2710434bcabffddc1d8e3ed215"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "c2b272e5591b981c636e7b5f7aca37b5"
server_key_iv: "f1a25a640c440ad384c53bce32c51d60"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.320165 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:16:49.322618 12831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.323956 12816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 15:16:49.324095 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "65524d2710434bcabffddc1d8e3ed215"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "c2b272e5591b981c636e7b5f7aca37b5"
server_key_iv: "f1a25a640c440ad384c53bce32c51d60"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.324237 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:49.339322 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:49.342294 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:49.342470 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:49.343046 12816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:49.343875 12816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:49.343922 12816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.343976 12816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:49.344004 12816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.352356 12816 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:40103
I20260927 15:16:49.352401 12944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:40103 every 8 connection(s)
I20260927 15:16:49.353223 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:49.356338 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 12816
I20260927 15:16:49.356446 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:49.356758 10186 external_mini_cluster.cc:1428] Setting key e89858cf7331b2364944517550e01d9f
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[12575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:16:49.366473 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.355035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:41887 (local address 127.9.242.190:39803)
0927 15:16:49.355153 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.355157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.356183 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 15:16:49.357301 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.357304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.357352 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.357417 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:49.358777 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.359341 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.360512 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.360737 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.360822 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.363113 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:49.363116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:49.363118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:49.363141 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:49.364527 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.365009 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.365012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.365013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.365084 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.365402 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.365405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.365407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.365606 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:49.365711 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.366198 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.366314 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:16:49.367362 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.355387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39803 (local address 127.9.242.130:41887)
0927 15:16:49.356009 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.356048 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:49.357077 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:49.357589 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:49.357596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:49.358004 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:49.358639 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.358658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.359454 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.359458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.360396 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.360405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.360872 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.360876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.361033 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.361560 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:49.361576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:49.362957 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:49.364659 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:49.364664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:49.364679 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:49.364913 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:49.365176 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:49.365179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:49.365180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:49.365314 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:49.365722 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:49.365729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:49.365968 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 15:16:49.366141 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 15:16:49.366452 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:16:49.368588 12945 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39803
I20260927 15:16:49.368821 12945 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:49.369290 12945 heartbeater.cc:511] Master 127.9.242.190:39803 requested a full tablet report, sending...
I20260927 15:16:49.370337 12617 ts_manager.cc:194] Registered new tserver with Master: 65524d2710434bcabffddc1d8e3ed215 (127.9.242.130:40103)
I20260927 15:16:49.371075 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:41887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:49.385090 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39803
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:49.494680 12950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:49.494954 12950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:49.494997 12950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:49.499056 12950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:49.499132 12950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:49.499190 12950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:49.504577 12950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:49.505550 12950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:49.506523 12950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:49.516040 12958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:49.516074 12956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:49.516078 12955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:49.516378 12950 server_base.cc:1034] running on GCE node
I20260927 15:16:49.516891 12950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:49.517498 12950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:49.518688 12950 hybrid_clock.cc:648] HybridClock initialized: now 1790522209518667 us; error 35 us; skew 500 ppm
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[12575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:49.521430 12950 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:49.522358 12950 webserver.cc:460] Webserver started at http://127.9.242.131:33065/ using document root <none> and password file <none>
I20260927 15:16:49.522879 12950 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:49.522934 12950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:49.523092 12950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:49.525060 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "c248052bb3a3400aa671701446cd37ed"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "024a7f791686b379c3a7c38c99427670"
server_key_iv: "140c937f106c9c6e0b1f69047b3022de"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.525532 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "c248052bb3a3400aa671701446cd37ed"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "024a7f791686b379c3a7c38c99427670"
server_key_iv: "140c937f106c9c6e0b1f69047b3022de"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.529198 12950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 15:16:49.531563 12965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.532793 12950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:49.532907 12950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "c248052bb3a3400aa671701446cd37ed"
format_stamp: "Formatted at 2026-09-27 15:16:49 on dist-test-slave-m4lt"
server_key: "024a7f791686b379c3a7c38c99427670"
server_key_iv: "140c937f106c9c6e0b1f69047b3022de"
server_key_version: "encryptionkey@0"
I20260927 15:16:49.533007 12950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:49.564545 12950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:49.567830 12950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:49.568030 12950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:49.568610 12950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:49.569413 12950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:49.569461 12950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.569512 12950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:49.569540 12950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:49.577831 12950 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:41567
I20260927 15:16:49.577859 13078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:41567 every 8 connection(s)
I20260927 15:16:49.578711 12950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:49.581734 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 12950
I20260927 15:16:49.581851 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:49.582144 10186 external_mini_cluster.cc:1428] Setting key 286055533cac9953e98de9a6b3685c5a
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[12575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:49.591957 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.580444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:35379 (local address 127.9.242.190:39803)
0927 15:16:49.580578 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.580583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.581729 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 15:16:49.582752 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.582755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.582808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.582894 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:49.584348 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.584877 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.586184 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.586405 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.586486 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.588878 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:49.588882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:49.588885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:49.588911 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:49.590069 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.590565 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.590569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.590570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.590643 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:49.590941 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:49.590944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:49.590946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:49.591114 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:49.591192 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:49.591668 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 15:16:49.591797 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:16:49.592726 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:49.580816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39803 (local address 127.9.242.131:35379)
0927 15:16:49.581503 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:49.581559 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:49.582542 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:49.583078 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:49.583148 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:49.583603 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:49.584197 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.584214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.585095 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.585099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.586074 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:49.586083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:49.586552 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:49.586555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:49.586708 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:49.587254 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:49.587270 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:49.588723 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:49.590171 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:49.590177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:49.590239 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:49.590473 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:49.590725 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:49.590728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:49.590730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:49.590858 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:49.591216 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:49.591223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:49.591451 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 15:16:49.591614 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:16:49.591920 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":103,"threads_started":1}
I20260927 15:16:49.594002 13079 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39803
I20260927 15:16:49.594259 13079 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:49.594781 13079 heartbeater.cc:511] Master 127.9.242.190:39803 requested a full tablet report, sending...
I20260927 15:16:49.595736 12617 ts_manager.cc:194] Registered new tserver with Master: c248052bb3a3400aa671701446cd37ed (127.9.242.131:41567)
I20260927 15:16:49.596280 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:35379
I20260927 15:16:49.607658 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:16:49.619438 12659 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:16:49.617997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.9.242.190:39803)
0927 15:16:49.618137 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:49.618142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:49.618354 (+ 212us) server_negotiation.cc:407] Connection header received
0927 15:16:49.618482 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:49.618486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:49.618531 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:49.618603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:49.619305 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35074: BlockingRecv error: recv got EOF from 127.0.0.1:35074 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:16:49.620325 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 12682
I20260927 15:16:49.624759 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 12816
I20260927 15:16:49.628854 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 12950
I20260927 15:16:49.632773 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 12588
2026-09-27T15:16:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2022 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[13091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[13091](info): set up 2 sockets
Sep 27 15:16:49 dist-test-slave-m4lt krb5kdc[13091](info): commencing operation
krb5kdc: starting...
W20260927 15:16:50.593485 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:50 dist-test-slave-m4lt krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:50.717298 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39085
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:39085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:50.824994 13104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:50.825279 13104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:50.825320 13104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:50.829296 13104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:50.829361 13104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:50.829380 13104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:50.829396 13104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:50.829409 13104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:50.834693 13104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:39085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:50.835690 13104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:50.836628 13104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:50.844834 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:50.844848 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:50.844893 13104 server_base.cc:1034] running on GCE node
W20260927 15:16:50.844838 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:50.845583 13104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:50.846738 13104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:50.847899 13104 hybrid_clock.cc:648] HybridClock initialized: now 1790522210847875 us; error 39 us; skew 500 ppm
Sep 27 15:16:50 dist-test-slave-m4lt krb5kdc[13091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522210, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:50.850530 13104 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:50.851418 13104 webserver.cc:460] Webserver started at http://127.9.242.190:46391/ using document root <none> and password file <none>
I20260927 15:16:50.851933 13104 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:50.851994 13104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:50.852177 13104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:50.854218 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "091a74bea5324dc5bc5c0e1d88d3c99e"
format_stamp: "Formatted at 2026-09-27 15:16:50 on dist-test-slave-m4lt"
server_key: "bdc84455261ef91bf5c5bd9219acb371"
server_key_iv: "14b014cff44e4543274ae1f88d1e0a54"
server_key_version: "encryptionkey@0"
I20260927 15:16:50.854751 13104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "091a74bea5324dc5bc5c0e1d88d3c99e"
format_stamp: "Formatted at 2026-09-27 15:16:50 on dist-test-slave-m4lt"
server_key: "bdc84455261ef91bf5c5bd9219acb371"
server_key_iv: "14b014cff44e4543274ae1f88d1e0a54"
server_key_version: "encryptionkey@0"
I20260927 15:16:50.858551 13104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 15:16:50.860996 13119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:50.862617 13104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:16:50.862743 13104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "091a74bea5324dc5bc5c0e1d88d3c99e"
format_stamp: "Formatted at 2026-09-27 15:16:50 on dist-test-slave-m4lt"
server_key: "bdc84455261ef91bf5c5bd9219acb371"
server_key_iv: "14b014cff44e4543274ae1f88d1e0a54"
server_key_version: "encryptionkey@0"
I20260927 15:16:50.862844 13104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:50.893616 13104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:50.896737 13104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:50.896914 13104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:50.903439 13104 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:39085
I20260927 15:16:50.903698 13171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:39085 every 8 connection(s)
I20260927 15:16:50.904358 13104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:50.907130 13172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:50.912528 13172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e: Bootstrap starting.
I20260927 15:16:50.914208 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13104
I20260927 15:16:50.914311 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:50.914551 10186 external_mini_cluster.cc:1428] Setting key 97e26e7f0c34d331dfef97b83386995b
I20260927 15:16:50.915397 13172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:50.916177 13172 log.cc:824] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:50.918305 13172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e: No bootstrap required, opened a new log
I20260927 15:16:50.922252 13172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39085 } }
I20260927 15:16:50.922514 13172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:50.922600 13172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 091a74bea5324dc5bc5c0e1d88d3c99e, State: Initialized, Role: FOLLOWER
I20260927 15:16:50.923087 13172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39085 } }
I20260927 15:16:50.923267 13172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:50.923358 13172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:50.923460 13172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:50.924603 13172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39085 } }
I20260927 15:16:50.924988 13172 leader_election.cc:302] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 091a74bea5324dc5bc5c0e1d88d3c99e; no voters:
Sep 27 15:16:50 dist-test-slave-m4lt krb5kdc[13091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:50.925380 13172 leader_election.cc:288] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:50.925599 13177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:50.925987 13177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [term 1 LEADER]: Becoming Leader. State: Replica: 091a74bea5324dc5bc5c0e1d88d3c99e, State: Running, Role: LEADER
I20260927 15:16:50.926625 13177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39085 } }
I20260927 15:16:50.926837 13172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:50.928251 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39085 } } }
I20260927 15:16:50.928376 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:50.928620 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 091a74bea5324dc5bc5c0e1d88d3c99e. Latest consensus state: current_term: 1 leader_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "091a74bea5324dc5bc5c0e1d88d3c99e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39085 } } }
I20260927 15:16:50.928721 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:50.931253 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:50.915574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.9.242.190:39085)
0927 15:16:50.916049 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:50.916072 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:50.916113 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:16:50.916842 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:50.916864 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:50.917267 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:50.917622 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:50.918725 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:50.920184 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:50.921902 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:50.922579 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:50.922748 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:50.925834 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:50.925840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:50.925858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:50.925892 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:50.927936 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:50.928480 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:50.928486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:50.928492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:50.928638 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:50.928970 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:50.928973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:50.928975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:50.929424 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:50.929624 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:50.929970 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:16:50.930298 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":145,"threads_started":1}
W20260927 15:16:50.931520 13192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:16:50.931589 13192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:16:50.931666 13189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:50.932400 13189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:50.934695 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:16:50.938314 13189 catalog_manager.cc:1347] Generated new cluster ID: e33352a768894b1988039808c57cec80
I20260927 15:16:50.938387 13189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:50.950911 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:50.946299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.9.242.190:39085)
0927 15:16:50.946417 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:50.946422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:50.946638 (+ 216us) server_negotiation.cc:407] Connection header received
0927 15:16:50.946750 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:50.946754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:50.946800 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:50.946863 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:50.947362 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:50.947876 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:50.948711 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:50.948936 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:50.949020 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:50.949855 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:50.949859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:50.949862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:50.950423 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:50.950456 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:50.950476 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:50.950561 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:50.950715 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 15:16:50.950777 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:16:50.951256 13189 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:50.952185 13189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:50.956502 13189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 091a74bea5324dc5bc5c0e1d88d3c99e: Generated new TSK 0
I20260927 15:16:50.957278 13189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:16:50.962020 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13104
2026-09-27T15:16:50Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1327 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:51 dist-test-slave-m4lt krb5kdc[13205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:51 dist-test-slave-m4lt krb5kdc[13205](info): set up 2 sockets
Sep 27 15:16:51 dist-test-slave-m4lt krb5kdc[13205](info): commencing operation
krb5kdc: starting...
W20260927 15:16:51.933454 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:52.052603 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37287
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:52.162451 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:52.162712 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:52.162755 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:52.166747 13218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:52.166802 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:52.166819 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:52.166836 13218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:52.166863 13218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:52.172205 13218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:37287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:52.173209 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:52.174257 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:52.182581 13223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:52.182593 13226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:52.182655 13224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:52.182824 13218 server_base.cc:1034] running on GCE node
I20260927 15:16:52.183467 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:52.184613 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:52.185745 13218 hybrid_clock.cc:648] HybridClock initialized: now 1790522212185714 us; error 41 us; skew 500 ppm
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:52.188514 13218 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:52.189415 13218 webserver.cc:460] Webserver started at http://127.9.242.190:38851/ using document root <none> and password file <none>
I20260927 15:16:52.189970 13218 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:52.190028 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:52.190197 13218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:52.192126 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "aefe0025d52f45a6aa2a5a23c135e786"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "e30f1c2a9871ce77601eb05ce74fd4d0"
server_key_iv: "fad12db4c30cff44b417208c76c85f3d"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.192622 13218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "aefe0025d52f45a6aa2a5a23c135e786"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "e30f1c2a9871ce77601eb05ce74fd4d0"
server_key_iv: "fad12db4c30cff44b417208c76c85f3d"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.196334 13218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 15:16:52.198750 13233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.200066 13218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:52.200176 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "aefe0025d52f45a6aa2a5a23c135e786"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "e30f1c2a9871ce77601eb05ce74fd4d0"
server_key_iv: "fad12db4c30cff44b417208c76c85f3d"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.200266 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:52.216279 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:52.219547 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:52.219707 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:52.226802 13218 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:37287
I20260927 15:16:52.226830 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:37287 every 8 connection(s)
I20260927 15:16:52.227845 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:52.228150 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13218
I20260927 15:16:52.228253 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:52.228545 10186 external_mini_cluster.cc:1428] Setting key c9253600b25be45d4a349a76cd65fefa
I20260927 15:16:52.231043 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:52.236357 13286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786: Bootstrap starting.
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:52.238652 13286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:52.239318 13286 log.cc:824] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:52.241493 13286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786: No bootstrap required, opened a new log
I20260927 15:16:52.242650 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.229924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.9.242.190:37287)
0927 15:16:52.230354 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:52.230368 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:52.230404 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:16:52.231178 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:52.231211 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:52.231566 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:52.231879 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:52.232666 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.233714 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.234541 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.234992 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.235150 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.237603 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:52.237608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:52.237620 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:52.237646 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:52.239284 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.239704 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.239708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.239712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.239817 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.240129 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.240133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.240134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.240595 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:52.240781 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:52.241009 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 15:16:52.241310 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
I20260927 15:16:52.244984 13286 raft_consensus.cc:348] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe0025d52f45a6aa2a5a23c135e786" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37287 } }
I20260927 15:16:52.245186 13286 raft_consensus.cc:374] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:52.245220 13286 raft_consensus.cc:729] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe0025d52f45a6aa2a5a23c135e786, State: Initialized, Role: FOLLOWER
I20260927 15:16:52.245564 13286 consensus_queue.cc:260] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe0025d52f45a6aa2a5a23c135e786" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37287 } }
I20260927 15:16:52.245712 13286 raft_consensus.cc:388] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:52.245764 13286 raft_consensus.cc:482] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:52.245832 13286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:52.246610 13286 raft_consensus.cc:504] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe0025d52f45a6aa2a5a23c135e786" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37287 } }
I20260927 15:16:52.246884 13286 leader_election.cc:302] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aefe0025d52f45a6aa2a5a23c135e786; no voters:
I20260927 15:16:52.247118 13286 leader_election.cc:288] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:52.247241 13291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:52.247437 13291 raft_consensus.cc:686] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [term 1 LEADER]: Becoming Leader. State: Replica: aefe0025d52f45a6aa2a5a23c135e786, State: Running, Role: LEADER
I20260927 15:16:52.247701 13291 consensus_queue.cc:237] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe0025d52f45a6aa2a5a23c135e786" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37287 } }
I20260927 15:16:52.248178 13286 sys_catalog.cc:564] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:52.249279 13293 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aefe0025d52f45a6aa2a5a23c135e786. Latest consensus state: current_term: 1 leader_uuid: "aefe0025d52f45a6aa2a5a23c135e786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe0025d52f45a6aa2a5a23c135e786" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37287 } } }
I20260927 15:16:52.249396 13293 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:52.249724 13292 sys_catalog.cc:455] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aefe0025d52f45a6aa2a5a23c135e786" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aefe0025d52f45a6aa2a5a23c135e786" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37287 } } }
I20260927 15:16:52.249825 13292 sys_catalog.cc:458] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:52.249778 13300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:52.252804 13300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:52.257637 13300 catalog_manager.cc:1347] Generated new cluster ID: f7c469e0b01a4dc18a69e0df6f6232a6
I20260927 15:16:52.257725 13300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:52.271900 13300 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:52.272881 13300 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:52.287165 13300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aefe0025d52f45a6aa2a5a23c135e786: Generated new TSK 0
I20260927 15:16:52.287842 13300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:52.314917 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37287
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:52.425827 13312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:52.426103 13312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:52.426165 13312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:52.430166 13312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:52.430226 13312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:52.430310 13312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:52.435531 13312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:52.436579 13312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:52.437556 13312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:52.446846 13318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:52.446920 13312 server_base.cc:1034] running on GCE node
W20260927 15:16:52.446839 13317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:52.446844 13320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:52.447520 13312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:52.448216 13312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:52.449406 13312 hybrid_clock.cc:648] HybridClock initialized: now 1790522212449374 us; error 48 us; skew 500 ppm
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:52.452194 13312 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:52.453058 13312 webserver.cc:460] Webserver started at http://127.9.242.129:33265/ using document root <none> and password file <none>
I20260927 15:16:52.453581 13312 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:52.453644 13312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:52.453912 13312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:52.455863 13312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "8b900ec8f22842e8a67d89849c09b461"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "2385ffcb60ae4c046d05793048c373ba"
server_key_iv: "03ce93c3c3ae391265768d77b2180657"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.456382 13312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "8b900ec8f22842e8a67d89849c09b461"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "2385ffcb60ae4c046d05793048c373ba"
server_key_iv: "03ce93c3c3ae391265768d77b2180657"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.459928 13312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 15:16:52.462308 13327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.463704 13312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:52.463821 13312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "8b900ec8f22842e8a67d89849c09b461"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "2385ffcb60ae4c046d05793048c373ba"
server_key_iv: "03ce93c3c3ae391265768d77b2180657"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.463915 13312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:52.487083 13312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:52.490428 13312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:52.490597 13312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:52.491176 13312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:52.491959 13312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:52.492003 13312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.492054 13312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:52.492079 13312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.500377 13312 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:38737
I20260927 15:16:52.500427 13440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:38737 every 8 connection(s)
I20260927 15:16:52.501268 13312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:52.501431 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13312
I20260927 15:16:52.501534 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:52.501889 10186 external_mini_cluster.cc:1428] Setting key 09afd5e14a84662e472f531a62e95990
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:16:52.515439 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.503080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:56873 (local address 127.9.242.190:37287)
0927 15:16:52.503278 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:52.503282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:52.504275 (+ 993us) server_negotiation.cc:407] Connection header received
0927 15:16:52.505893 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:52.505896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:52.505940 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:52.506010 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:52.507131 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.507652 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.509227 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.509589 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.509762 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.512047 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:52.512051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:52.512054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:52.512082 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:52.513343 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.513865 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.513868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.513869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.513951 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.514386 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.514388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.514390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.514569 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:52.514644 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:52.515207 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 15:16:52.515315 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 15:16:52.516526 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.503588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37287 (local address 127.9.242.129:56873)
0927 15:16:52.504067 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:52.504127 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:52.505678 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:52.506207 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:52.506213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:52.506551 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:52.507012 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:52.507025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.507769 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.507772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:52.509102 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:52.509113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.509791 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.509794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:52.509936 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.510467 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:52.510484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:52.511886 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:52.513445 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:52.513449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:52.513462 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:52.513761 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:52.514150 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:52.514153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:52.514155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:52.514313 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:52.514691 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:52.514696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:52.514981 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 15:16:52.515163 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 15:16:52.515506 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260927 15:16:52.517835 13441 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37287
I20260927 15:16:52.518078 13441 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:52.518626 13441 heartbeater.cc:511] Master 127.9.242.190:37287 requested a full tablet report, sending...
I20260927 15:16:52.520171 13250 ts_manager.cc:194] Registered new tserver with Master: 8b900ec8f22842e8a67d89849c09b461 (127.9.242.129:38737)
I20260927 15:16:52.521328 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:56873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:52.530647 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37287
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:52.638479 13446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:52.638736 13446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:52.638790 13446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:52.642932 13446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:52.642997 13446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:52.643083 13446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:52.648424 13446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:52.649456 13446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:52.650466 13446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:52.659736 13454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:52.659762 13452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:52.659713 13451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:52.659833 13446 server_base.cc:1034] running on GCE node
I20260927 15:16:52.660404 13446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:52.661074 13446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:52.662251 13446 hybrid_clock.cc:648] HybridClock initialized: now 1790522212662229 us; error 35 us; skew 500 ppm
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:52.664932 13446 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:52.665822 13446 webserver.cc:460] Webserver started at http://127.9.242.130:42309/ using document root <none> and password file <none>
I20260927 15:16:52.666344 13446 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:52.666405 13446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:52.666601 13446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:52.668650 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "9b5d7ecc960244a487d0f16dbd674795"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "4f07da7137d2ef8ed988c0ed318a9527"
server_key_iv: "ee40abed0aa47fa82f1435475f2efc12"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.669188 13446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "9b5d7ecc960244a487d0f16dbd674795"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "4f07da7137d2ef8ed988c0ed318a9527"
server_key_iv: "ee40abed0aa47fa82f1435475f2efc12"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.672907 13446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:16:52.675526 13461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.676937 13446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:52.677062 13446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "9b5d7ecc960244a487d0f16dbd674795"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "4f07da7137d2ef8ed988c0ed318a9527"
server_key_iv: "ee40abed0aa47fa82f1435475f2efc12"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.677168 13446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:52.708021 13446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:52.711093 13446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:52.711283 13446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:52.711874 13446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:52.712694 13446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:52.712740 13446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.712792 13446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:52.712821 13446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.721247 13446 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:41823
I20260927 15:16:52.721276 13574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:41823 every 8 connection(s)
I20260927 15:16:52.722126 13446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:52.726788 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13446
I20260927 15:16:52.726897 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:52.727137 10186 external_mini_cluster.cc:1428] Setting key 652df05b1df8c5a4f3a2eac71ba0bf0d
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:52.735600 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.723798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:33339 (local address 127.9.242.190:37287)
0927 15:16:52.723955 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:52.723959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:52.724842 (+ 883us) server_negotiation.cc:407] Connection header received
0927 15:16:52.726223 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:52.726226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:52.726273 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:52.726350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:52.727616 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.728151 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.729326 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.729553 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.729640 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.731998 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:52.732001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:52.732003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:52.732026 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:52.733363 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.733962 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.733965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.733966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.734038 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.734389 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.734392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.734393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.734557 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:52.734625 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:52.735356 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 15:16:52.735466 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:16:52.736526 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.724256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37287 (local address 127.9.242.130:33339)
0927 15:16:52.724684 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:52.724729 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:52.725993 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:52.726515 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:52.726521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:52.726940 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:52.727472 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:52.727485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.728279 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.728282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:52.729213 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:52.729222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.729715 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.729720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:52.729879 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.730414 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:52.730430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:52.731835 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:52.733478 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:52.733485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:52.733502 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:52.733873 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:52.734121 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:52.734124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:52.734126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:52.734306 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:52.734657 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:52.734663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:52.734986 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 15:16:52.735215 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 15:16:52.735571 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260927 15:16:52.737747 13575 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37287
I20260927 15:16:52.737996 13575 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:16:52.738663 13575 heartbeater.cc:511] Master 127.9.242.190:37287 requested a full tablet report, sending...
I20260927 15:16:52.739809 13250 ts_manager.cc:194] Registered new tserver with Master: 9b5d7ecc960244a487d0f16dbd674795 (127.9.242.130:41823)
I20260927 15:16:52.740381 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:33339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:52.752759 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37287
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:41943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:52.859328 13580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:52.859584 13580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:52.859650 13580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:52.863662 13580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:52.863729 13580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:52.863807 13580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:52.869154 13580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:52.870217 13580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:52.871202 13580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:52.880744 13585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:52.880748 13586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:52.880796 13580 server_base.cc:1034] running on GCE node
W20260927 15:16:52.880744 13588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:52.881397 13580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:52.882064 13580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:52.883227 13580 hybrid_clock.cc:648] HybridClock initialized: now 1790522212883215 us; error 32 us; skew 500 ppm
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:52.885952 13580 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:52.886780 13580 webserver.cc:460] Webserver started at http://127.9.242.131:37569/ using document root <none> and password file <none>
I20260927 15:16:52.887305 13580 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:52.887356 13580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:52.887531 13580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:52.889637 13580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "c81c6f0d080543609643ac373d39300f"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "8bfe6d0422eef20b67b6cc3d5f5557a6"
server_key_iv: "ecc07f6059a160199fbfe07bf12a3894"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.890192 13580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "c81c6f0d080543609643ac373d39300f"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "8bfe6d0422eef20b67b6cc3d5f5557a6"
server_key_iv: "ecc07f6059a160199fbfe07bf12a3894"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.893702 13580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:16:52.896001 13595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.897183 13580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:52.897284 13580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "c81c6f0d080543609643ac373d39300f"
format_stamp: "Formatted at 2026-09-27 15:16:52 on dist-test-slave-m4lt"
server_key: "8bfe6d0422eef20b67b6cc3d5f5557a6"
server_key_iv: "ecc07f6059a160199fbfe07bf12a3894"
server_key_version: "encryptionkey@0"
I20260927 15:16:52.897383 13580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:52.926455 13580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:52.929540 13580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:52.929726 13580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:52.930316 13580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:52.931143 13580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:52.931187 13580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.931227 13580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:52.931255 13580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:52.939847 13580 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:36361
I20260927 15:16:52.939879 13708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:36361 every 8 connection(s)
I20260927 15:16:52.940707 13580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:52.948493 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13580
I20260927 15:16:52.948614 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:52.948879 10186 external_mini_cluster.cc:1428] Setting key a1d4472e08c4d8214d9ce617757f7d8c
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:52.953568 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.942388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:46065 (local address 127.9.242.190:37287)
0927 15:16:52.942538 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:52.942543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:52.943379 (+ 836us) server_negotiation.cc:407] Connection header received
0927 15:16:52.944668 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:52.944672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:52.944717 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:52.944786 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:52.945959 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.946511 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.947603 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.947808 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.947879 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.950495 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:52.950498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:52.950501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:52.950527 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:52.951662 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.952079 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.952082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.952083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.952146 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.952518 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.952522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.952524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.952712 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:52.952806 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:52.953362 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 15:16:52.953456 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:16:52.954381 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.942785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37287 (local address 127.9.242.131:46065)
0927 15:16:52.943230 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:52.943273 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:52.944481 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:52.944927 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:52.944933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:52.945262 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:52.945818 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:52.945834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.946616 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.946619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:52.947499 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:52.947508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.947914 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.947917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:52.948055 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.948759 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:52.948774 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:52.950353 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:52.951756 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:52.951760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:52.951772 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:52.952008 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:52.952229 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:52.952231 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:52.952232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:52.952355 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:52.952832 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:52.952838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:52.953074 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 15:16:52.953307 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 15:16:52.953563 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260927 15:16:52.955436 13709 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37287
I20260927 15:16:52.955657 13709 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:52.956182 13709 heartbeater.cc:511] Master 127.9.242.190:37287 requested a full tablet report, sending...
I20260927 15:16:52.957086 13250 ts_manager.cc:194] Registered new tserver with Master: c81c6f0d080543609643ac373d39300f (127.9.242.131:36361)
I20260927 15:16:52.957615 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:46065
I20260927 15:16:52.962512 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 15:16:52 dist-test-slave-m4lt krb5kdc[13205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:52.985935 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:52.978283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.9.242.190:37287)
0927 15:16:52.978399 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:52.978402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:52.978451 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:16:52.978576 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:52.978579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:52.978622 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:52.978685 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:52.979334 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.979887 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.980794 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:52.981015 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:52.981088 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:52.983343 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:52.983346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:52.983348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:52.983371 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:52.984626 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.984960 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.984963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.984964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.985035 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:52.985219 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:52.985220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:52.985222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:52.985427 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:52.985521 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:52.985723 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 15:16:52.985816 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:16:52.986308 13250 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48188
I20260927 15:16:52.987069 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13312
I20260927 15:16:52.991958 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13446
I20260927 15:16:52.996151 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13580
I20260927 15:16:53.000398 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13218
2026-09-27T15:16:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2040 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:53 dist-test-slave-m4lt krb5kdc[13721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:53 dist-test-slave-m4lt krb5kdc[13721](info): set up 2 sockets
Sep 27 15:16:53 dist-test-slave-m4lt krb5kdc[13721](info): commencing operation
krb5kdc: starting...
W20260927 15:16:53.961366 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:54.079911 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:38399
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:38399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:54.188400 13734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:54.188681 13734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:54.188737 13734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:54.192736 13734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:54.192800 13734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:54.192826 13734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:54.192847 13734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:54.192867 13734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:54.198199 13734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:38399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:38399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:54.199226 13734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:54.200197 13734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:54.208488 13739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:54.208546 13734 server_base.cc:1034] running on GCE node
W20260927 15:16:54.208478 13742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:54.208496 13740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:54.209237 13734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:54.210444 13734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:54.211689 13734 hybrid_clock.cc:648] HybridClock initialized: now 1790522214211663 us; error 40 us; skew 500 ppm
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:54.214504 13734 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:54.215436 13734 webserver.cc:460] Webserver started at http://127.9.242.190:43297/ using document root <none> and password file <none>
I20260927 15:16:54.215965 13734 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:54.216028 13734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:54.216241 13734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:54.218382 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "00b496158ace4dd99cc57f8b71f4ded7"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "8d8fcaca4e25ad9c6af2999c212f5ffa"
server_key_iv: "2037b7f1b01bdb82a37b75d0ce04490c"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.218919 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "00b496158ace4dd99cc57f8b71f4ded7"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "8d8fcaca4e25ad9c6af2999c212f5ffa"
server_key_iv: "2037b7f1b01bdb82a37b75d0ce04490c"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.222654 13734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:54.225287 13749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.226694 13734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:54.226814 13734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "00b496158ace4dd99cc57f8b71f4ded7"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "8d8fcaca4e25ad9c6af2999c212f5ffa"
server_key_iv: "2037b7f1b01bdb82a37b75d0ce04490c"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.226915 13734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:54.251674 13734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:54.254894 13734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:54.255067 13734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:54.261595 13734 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:38399
I20260927 15:16:54.261636 13801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:38399 every 8 connection(s)
I20260927 15:16:54.262543 13734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:54.265241 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:54.265580 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13734
I20260927 15:16:54.265735 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:54.265990 10186 external_mini_cluster.cc:1428] Setting key a7a5e0e0640f87b640d8b3b60b0575d0
I20260927 15:16:54.271142 13802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7: Bootstrap starting.
I20260927 15:16:54.273720 13802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7: Neither blocks nor log segments found. Creating new log.
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:54.274577 13802 log.cc:824] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:54.276454 13802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7: No bootstrap required, opened a new log
I20260927 15:16:54.279641 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:54.267405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.9.242.190:38399)
0927 15:16:54.267775 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:54.267789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:54.267823 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:16:54.268439 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:54.268455 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:54.268792 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:54.269052 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:54.269906 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.270852 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.271932 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.272380 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.272492 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:54.275212 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:54.275218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:54.275235 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:54.275264 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:54.276864 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.277330 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.277335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.277339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.277443 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.277739 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.277742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.277743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.278159 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:54.278308 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:54.278645 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 15:16:54.278914 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":93,"threads_started":1}
I20260927 15:16:54.280145 13802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b496158ace4dd99cc57f8b71f4ded7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38399 } }
I20260927 15:16:54.280357 13802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:54.280417 13802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b496158ace4dd99cc57f8b71f4ded7, State: Initialized, Role: FOLLOWER
I20260927 15:16:54.280880 13802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b496158ace4dd99cc57f8b71f4ded7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38399 } }
I20260927 15:16:54.281032 13802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:54.281134 13802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:54.281237 13802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:54.282240 13802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b496158ace4dd99cc57f8b71f4ded7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38399 } }
I20260927 15:16:54.282532 13802 leader_election.cc:302] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00b496158ace4dd99cc57f8b71f4ded7; no voters:
I20260927 15:16:54.282770 13802 leader_election.cc:288] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:54.282868 13807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:54.283103 13807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [term 1 LEADER]: Becoming Leader. State: Replica: 00b496158ace4dd99cc57f8b71f4ded7, State: Running, Role: LEADER
I20260927 15:16:54.283438 13807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b496158ace4dd99cc57f8b71f4ded7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38399 } }
I20260927 15:16:54.283850 13802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:54.284932 13808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00b496158ace4dd99cc57f8b71f4ded7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b496158ace4dd99cc57f8b71f4ded7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38399 } } }
I20260927 15:16:54.285070 13808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:54.286221 13809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00b496158ace4dd99cc57f8b71f4ded7. Latest consensus state: current_term: 1 leader_uuid: "00b496158ace4dd99cc57f8b71f4ded7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00b496158ace4dd99cc57f8b71f4ded7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38399 } } }
I20260927 15:16:54.286296 13809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7 [sys.catalog]: This master's current role is: LEADER
W20260927 15:16:54.288425 13822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:16:54.288496 13822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:16:54.288563 13816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:54.289350 13816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:54.294667 13816 catalog_manager.cc:1347] Generated new cluster ID: dd490ea796ca486eb00bc279214bf38e
I20260927 15:16:54.294728 13816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:54.303503 13816 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:54.304740 13816 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:54.310024 13816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00b496158ace4dd99cc57f8b71f4ded7: Generated new TSK 0
I20260927 15:16:54.310791 13816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:54.348737 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38399
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:54.457741 13828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:54.458050 13828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:54.458097 13828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:54.462332 13828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:54.462397 13828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:54.462457 13828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:54.467852 13828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:54.469053 13828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:54.470122 13828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:54.479059 13836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:54.479202 13828 server_base.cc:1034] running on GCE node
W20260927 15:16:54.479167 13834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:54.479086 13833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:54.479735 13828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:54.480386 13828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:54.481549 13828 hybrid_clock.cc:648] HybridClock initialized: now 1790522214481527 us; error 35 us; skew 500 ppm
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:54.484382 13828 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:54.485353 13828 webserver.cc:460] Webserver started at http://127.9.242.129:32849/ using document root <none> and password file <none>
I20260927 15:16:54.485975 13828 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:54.486047 13828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:54.486222 13828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:54.488224 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "79ad1674b59c4b9fb0036a97bd9f810f"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "53c7bc3f3e58cab0728fc833c7b36b41"
server_key_iv: "9dd01a0462d0dcb0ff9b6fafc549fedc"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.488734 13828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "79ad1674b59c4b9fb0036a97bd9f810f"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "53c7bc3f3e58cab0728fc833c7b36b41"
server_key_iv: "9dd01a0462d0dcb0ff9b6fafc549fedc"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.492357 13828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:16:54.494683 13843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.495884 13828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:16:54.495996 13828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "79ad1674b59c4b9fb0036a97bd9f810f"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "53c7bc3f3e58cab0728fc833c7b36b41"
server_key_iv: "9dd01a0462d0dcb0ff9b6fafc549fedc"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.496170 13828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:54.513645 13828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:54.516726 13828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:54.516898 13828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:54.517452 13828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:54.518285 13828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:54.518332 13828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.518384 13828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:54.518410 13828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.527181 13828 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:36117
I20260927 15:16:54.527227 13956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:36117 every 8 connection(s)
I20260927 15:16:54.528056 13828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:54.534957 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13828
I20260927 15:16:54.535089 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:54.535338 10186 external_mini_cluster.cc:1428] Setting key 79ed96151472e09a58a5e219ed99416b
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:54.541898 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:54.529836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:53669 (local address 127.9.242.190:38399)
0927 15:16:54.529990 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:54.529993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:54.530896 (+ 903us) server_negotiation.cc:407] Connection header received
0927 15:16:54.532256 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:54.532259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:54.532307 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:54.532389 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:54.533590 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.534266 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.535482 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.535796 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.535906 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:54.538505 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:54.538509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:54.538512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:54.538547 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:54.539991 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.540436 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.540440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.540442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.540525 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.540821 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.540824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.540826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.541038 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:54.541133 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:54.541578 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 15:16:54.541732 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:16:54.542626 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:54.530273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38399 (local address 127.9.242.129:53669)
0927 15:16:54.530740 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:54.530789 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:54.532034 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:54.532598 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:54.532606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:54.532956 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:54.533447 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:54.533461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.534389 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.534392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:54.535370 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:54.535379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.535975 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.535979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:54.536155 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:54.536703 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:54.536721 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:54.538345 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:54.540099 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:54.540104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:54.540115 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:54.540344 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:54.540619 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:54.540621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:54.540622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:54.540744 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:54.541134 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:54.541139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:54.541355 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 15:16:54.541502 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:16:54.541793 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":104,"threads_started":1}
I20260927 15:16:54.543812 13957 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38399
I20260927 15:16:54.544044 13957 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:16:54.544514 13957 heartbeater.cc:511] Master 127.9.242.190:38399 requested a full tablet report, sending...
I20260927 15:16:54.546005 13766 ts_manager.cc:194] Registered new tserver with Master: 79ad1674b59c4b9fb0036a97bd9f810f (127.9.242.129:36117)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:16:54.547350 13766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:53669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:54.560820 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38399
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:54.667603 13962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:54.667872 13962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:54.667919 13962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:54.672088 13962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:54.672153 13962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:54.672217 13962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:54.677433 13962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:54.678509 13962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:54.679446 13962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:54.688745 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:54.688759 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:54.688764 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:54.688910 13962 server_base.cc:1034] running on GCE node
I20260927 15:16:54.689296 13962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:54.689931 13962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:54.691100 13962 hybrid_clock.cc:648] HybridClock initialized: now 1790522214691076 us; error 37 us; skew 500 ppm
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:54.693845 13962 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:54.694698 13962 webserver.cc:460] Webserver started at http://127.9.242.130:44775/ using document root <none> and password file <none>
I20260927 15:16:54.695207 13962 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:54.695271 13962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:54.695461 13962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:54.697427 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "f67594479155418ba90d299058126ca3"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "957578b2a589c50e439d220ee1fb9251"
server_key_iv: "473cb22112f6417fb063bef2b3b7073b"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.698057 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "f67594479155418ba90d299058126ca3"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "957578b2a589c50e439d220ee1fb9251"
server_key_iv: "473cb22112f6417fb063bef2b3b7073b"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.701949 13962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:16:54.705137 13977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.706460 13962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:16:54.706573 13962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "f67594479155418ba90d299058126ca3"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "957578b2a589c50e439d220ee1fb9251"
server_key_iv: "473cb22112f6417fb063bef2b3b7073b"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.706676 13962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:54.727197 13962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:54.730228 13962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:54.730399 13962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:54.730973 13962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:54.731880 13962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:54.731926 13962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.731977 13962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:54.732005 13962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.740131 13962 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:36987
I20260927 15:16:54.740201 14090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:36987 every 8 connection(s)
I20260927 15:16:54.740998 13962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:54.746822 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 13962
I20260927 15:16:54.746932 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:54.747180 10186 external_mini_cluster.cc:1428] Setting key bf5f52988fa3ef2469b70824cbd1b87b
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:54.754307 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:54.742721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:35943 (local address 127.9.242.190:38399)
0927 15:16:54.742851 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:54.742855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:54.743883 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 15:16:54.744802 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:54.744806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:54.744860 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:54.744932 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:54.746119 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.746673 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.747983 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.748299 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.748388 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:54.750981 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:54.750985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:54.750987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:54.751009 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:54.752219 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.752793 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.752795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.752796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.752862 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.753227 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.753229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.753230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.753384 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:54.753453 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:54.754054 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 15:16:54.754191 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:16:54.755407 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:54.743120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38399 (local address 127.9.242.130:35943)
0927 15:16:54.743739 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:54.743781 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:54.744607 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:54.745113 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:54.745120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:54.745470 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:54.745986 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:54.746000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.746807 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.746810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:54.747886 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:54.747896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.748455 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.748459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:54.748630 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:54.749186 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:54.749204 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:54.750831 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:54.752339 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:54.752346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:54.752361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:54.752703 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:54.752946 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:54.752950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:54.752951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:54.753143 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:54.753474 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:54.753481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:54.753817 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 15:16:54.754032 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 15:16:54.754373 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:16:54.756556 14091 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38399
I20260927 15:16:54.756783 14091 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:54.757244 14091 heartbeater.cc:511] Master 127.9.242.190:38399 requested a full tablet report, sending...
I20260927 15:16:54.758220 13766 ts_manager.cc:194] Registered new tserver with Master: f67594479155418ba90d299058126ca3 (127.9.242.130:36987)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:16:54.758833 13766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:35943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:54.773020 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38399
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:43481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:54.883991 14096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:54.884287 14096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:54.884331 14096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:54.889091 14096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:54.889160 14096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:54.889230 14096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:54.895005 14096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:54.896106 14096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:54.897156 14096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:54.907333 14101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:54.907485 14096 server_base.cc:1034] running on GCE node
W20260927 15:16:54.907348 14104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:54.907336 14102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:54.907965 14096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:54.908612 14096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:54.909749 14096 hybrid_clock.cc:648] HybridClock initialized: now 1790522214909730 us; error 35 us; skew 500 ppm
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:54.912639 14096 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:54.913599 14096 webserver.cc:460] Webserver started at http://127.9.242.131:37893/ using document root <none> and password file <none>
I20260927 15:16:54.914176 14096 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:54.914242 14096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:54.914441 14096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:54.916474 14096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "dcf28f3f6aeb413a996736dae46d1351"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "45aaf08f8e87a62ba0ac38552e039d23"
server_key_iv: "aea9aaef07fd005c99ee45669fc7ccc0"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.917001 14096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "dcf28f3f6aeb413a996736dae46d1351"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "45aaf08f8e87a62ba0ac38552e039d23"
server_key_iv: "aea9aaef07fd005c99ee45669fc7ccc0"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.921998 14096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 15:16:54.925148 14111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.926568 14096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:16:54.926685 14096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "dcf28f3f6aeb413a996736dae46d1351"
format_stamp: "Formatted at 2026-09-27 15:16:54 on dist-test-slave-m4lt"
server_key: "45aaf08f8e87a62ba0ac38552e039d23"
server_key_iv: "aea9aaef07fd005c99ee45669fc7ccc0"
server_key_version: "encryptionkey@0"
I20260927 15:16:54.926786 14096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:54.955120 14096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:54.958045 14096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:54.958201 14096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:54.958735 14096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:54.959504 14096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:54.959548 14096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.959594 14096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:54.959613 14096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:54.967783 14096 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:41385
I20260927 15:16:54.967818 14224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:41385 every 8 connection(s)
I20260927 15:16:54.968631 14096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:54.969270 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 14096
I20260927 15:16:54.969378 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:54.969698 10186 external_mini_cluster.cc:1428] Setting key 6f80daa5a4ad8c018a86127f0429b709
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:54.981307 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:54.970429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:44921 (local address 127.9.242.190:38399)
0927 15:16:54.970550 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:54.970554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:54.971758 (+ 1204us) server_negotiation.cc:407] Connection header received
0927 15:16:54.972675 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:54.972678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:54.972728 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:54.972800 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:54.973994 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.974514 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.975597 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.975803 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.975888 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:54.978223 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:54.978225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:54.978227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:54.978249 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:54.979454 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.979928 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.979931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.979932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.980002 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:54.980292 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:54.980296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:54.980298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:54.980464 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:54.980546 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:54.981004 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 15:16:54.981136 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:16:54.982160 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:54.970934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38399 (local address 127.9.242.131:44921)
0927 15:16:54.971627 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:54.971681 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:54.972457 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:54.972978 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:54.972985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:54.973351 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:54.973869 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:54.973884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.974628 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.974630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:54.975484 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:54.975492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:54.975931 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:54.975936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:54.976094 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:54.976614 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:54.976630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:54.978003 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:54.979556 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:54.979561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:54.979610 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:54.979838 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:54.980079 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:54.980082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:54.980085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:54.980210 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:54.980555 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:54.980562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:54.980792 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 15:16:54.980950 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:16:54.981280 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":131,"threads_started":1}
I20260927 15:16:54.983176 14225 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38399
I20260927 15:16:54.983392 14225 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:54.983847 14225 heartbeater.cc:511] Master 127.9.242.190:38399 requested a full tablet report, sending...
I20260927 15:16:54.984803 13766 ts_manager.cc:194] Registered new tserver with Master: dcf28f3f6aeb413a996736dae46d1351 (127.9.242.131:41385)
I20260927 15:16:54.985333 13766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:44921
I20260927 15:16:54.995302 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 15:16:54 dist-test-slave-m4lt krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:16:55 dist-test-slave-m4lt krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:16:55.021576 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:55.011575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.9.242.129:36117)
0927 15:16:55.011900 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:55.011915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:55.011939 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:16:55.011988 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:55.011993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:55.012245 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:55.012369 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:55.013073 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:55.013990 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:55.014827 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:55.015203 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:55.015279 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:55.017767 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:55.017777 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:55.017791 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:55.017817 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:55.019422 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:55.019907 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:55.019914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:55.019916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:55.020025 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:55.020387 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:55.020390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:55.020392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:55.020734 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:55.020970 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:55.021180 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 15:16:55.021344 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":139,"threads_started":1}
Sep 27 15:16:55 dist-test-slave-m4lt krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:55.032553 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:55.024730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.9.242.190:38399)
0927 15:16:55.024854 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:55.024858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:55.024872 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:16:55.025002 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:55.025006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:55.025051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:55.025114 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:55.025728 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:55.026240 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:55.026976 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:55.027199 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:55.027281 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:55.029512 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:55.029516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:55.029519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:55.029542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:55.030947 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:55.031335 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:55.031339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:55.031341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:55.031424 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:55.031625 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:55.031627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:55.031628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:55.031828 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:55.031931 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:55.032271 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:16:55.032372 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:16:55.032897 13766 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50970
I20260927 15:16:55.033474 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13828
I20260927 15:16:55.038425 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13962
I20260927 15:16:55.042943 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 14096
I20260927 15:16:55.047158 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 13734
2026-09-27T15:16:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2046 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:16:55 dist-test-slave-m4lt krb5kdc[14237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:16:55 dist-test-slave-m4lt krb5kdc[14237](info): set up 2 sockets
Sep 27 15:16:55 dist-test-slave-m4lt krb5kdc[14237](info): commencing operation
krb5kdc: starting...
W20260927 15:16:56.005695 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:16:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:16:56.120360 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:38683
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:38683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:56.227058 14250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:56.227363 14250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:56.227422 14250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:56.231622 14250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:56.231700 14250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:56.231729 14250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:56.231747 14250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:56.231768 14250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:56.237150 14250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:38683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:38683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:56.238302 14250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:56.239307 14250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:56.247172 14255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:56.247224 14258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:56.247244 14256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:56.247335 14250 server_base.cc:1034] running on GCE node
I20260927 15:16:56.248001 14250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:56.249075 14250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:56.250226 14250 hybrid_clock.cc:648] HybridClock initialized: now 1790522216250211 us; error 24 us; skew 500 ppm
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:56.258188 14250 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:16:56.258962 14250 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:56.259032 14250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:56.259220 14250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:56.261302 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "8b3ae4e8f66847c2ad34857729fe04b1"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "c8db76284828f6085850129ede5e7328"
server_key_iv: "50edfa5f0df4d5117d94ee5e463cf28a"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.261873 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "8b3ae4e8f66847c2ad34857729fe04b1"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "c8db76284828f6085850129ede5e7328"
server_key_iv: "50edfa5f0df4d5117d94ee5e463cf28a"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.265929 14250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 15:16:56.268292 14264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.269507 14250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:56.269646 14250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "8b3ae4e8f66847c2ad34857729fe04b1"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "c8db76284828f6085850129ede5e7328"
server_key_iv: "50edfa5f0df4d5117d94ee5e463cf28a"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.269773 14250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:56.281745 14250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:56.284480 14250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:56.284654 14250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:56.291216 14250 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:38683
I20260927 15:16:56.291517 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:38683 every 8 connection(s)
I20260927 15:16:56.291844 14250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:56.294585 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:56.295881 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 14250
I20260927 15:16:56.296006 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:56.296300 10186 external_mini_cluster.cc:1428] Setting key e2f15c026202dc22727a38b4f4745902
I20260927 15:16:56.300657 14317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1: Bootstrap starting.
I20260927 15:16:56.303197 14317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:56.303890 14317 log.cc:824] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:56.305902 14317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1: No bootstrap required, opened a new log
I20260927 15:16:56.309226 14317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38683 } }
I20260927 15:16:56.309437 14317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:56.309500 14317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3ae4e8f66847c2ad34857729fe04b1, State: Initialized, Role: FOLLOWER
I20260927 15:16:56.309715 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.297623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.9.242.190:38683)
0927 15:16:56.298027 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:56.298043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:56.298079 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:16:56.298707 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:56.298723 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:56.299046 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:56.299325 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:56.300139 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.301134 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.302265 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.302753 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.302870 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:56.305435 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:56.305440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:56.305454 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:56.305479 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:56.307026 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.307415 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.307421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.307427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.307529 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.307768 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.307771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.307773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.308145 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:56.308333 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:56.308578 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 15:16:56.308850 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260927 15:16:56.309976 14317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38683 } }
I20260927 15:16:56.310101 14317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:56.310158 14317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:56.310249 14317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:56.311264 14317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38683 } }
I20260927 15:16:56.311599 14317 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b3ae4e8f66847c2ad34857729fe04b1; no voters:
I20260927 15:16:56.311928 14317 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:56.311996 14322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:56.312178 14322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3ae4e8f66847c2ad34857729fe04b1, State: Running, Role: LEADER
I20260927 15:16:56.312453 14322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38683 } }
I20260927 15:16:56.313146 14317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:56.314121 14324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3ae4e8f66847c2ad34857729fe04b1. Latest consensus state: current_term: 1 leader_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38683 } } }
I20260927 15:16:56.314261 14324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:56.314697 14323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3ae4e8f66847c2ad34857729fe04b1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 38683 } } }
I20260927 15:16:56.314790 14323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1 [sys.catalog]: This master's current role is: LEADER
W20260927 15:16:56.317636 14337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:16:56.317744 14337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:16:56.317804 14331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:56.318558 14331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:56.323400 14331 catalog_manager.cc:1347] Generated new cluster ID: f6bbe98bf1324b818ba06cd7043bbb89
I20260927 15:16:56.323465 14331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:56.330269 14331 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:56.331496 14331 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:56.337456 14331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1: Generated new TSK 0
I20260927 15:16:56.338253 14331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:16:56.373449 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38683
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:56.481251 14343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:56.481509 14343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:56.481573 14343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:56.485630 14343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:56.485728 14343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:56.485816 14343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:56.491214 14343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:56.492278 14343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:56.493294 14343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:56.502282 14349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:56.502306 14348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:56.502302 14351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:56.502677 14343 server_base.cc:1034] running on GCE node
I20260927 15:16:56.503044 14343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:56.503618 14343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:56.504810 14343 hybrid_clock.cc:648] HybridClock initialized: now 1790522216504799 us; error 34 us; skew 500 ppm
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:56.507478 14343 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:16:56.508196 14343 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:56.508257 14343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:56.508448 14343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:56.510473 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "d221fbeff04bf467eeeffc6611584998"
server_key_iv: "c132da0e6b03505562005cae6d55b5a2"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.510994 14343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "d221fbeff04bf467eeeffc6611584998"
server_key_iv: "c132da0e6b03505562005cae6d55b5a2"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.514696 14343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:56.517009 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.518313 14343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:56.518419 14343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "d221fbeff04bf467eeeffc6611584998"
server_key_iv: "c132da0e6b03505562005cae6d55b5a2"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.518527 14343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:56.558131 14343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:56.561224 14343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:56.561388 14343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:56.561990 14343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:56.562794 14343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:56.562837 14343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.562879 14343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:56.562906 14343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.570983 14343 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:46107
I20260927 15:16:56.571012 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:46107 every 8 connection(s)
I20260927 15:16:56.571511 14343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:56.579949 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 14343
I20260927 15:16:56.580078 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:56.580394 10186 external_mini_cluster.cc:1428] Setting key f80bd1c5da61de4dc4c5d64c3b7263b2
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:56.584801 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.573247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:35371 (local address 127.9.242.190:38683)
0927 15:16:56.573381 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:56.573385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:56.574303 (+ 918us) server_negotiation.cc:407] Connection header received
0927 15:16:56.575513 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:56.575516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:56.575569 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:56.575642 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:56.576788 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.577318 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.578477 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.578715 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.578801 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:56.581336 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:56.581339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:56.581341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:56.581364 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:56.582900 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.583401 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.583405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.583407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.583493 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.583812 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.583816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.583818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.583991 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:56.584084 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:56.584582 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 15:16:56.584684 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:16:56.586004 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.573561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38683 (local address 127.9.242.129:35371)
0927 15:16:56.574146 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:56.574190 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:56.575307 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:56.575814 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:56.575821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:56.576176 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:56.576646 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:56.576660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.577433 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.577436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:56.578362 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:56.578372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.578849 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.578853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:56.579024 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:56.579521 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:56.579537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:56.581188 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:56.583014 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:56.583019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:56.583031 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:56.583299 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:56.583576 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:56.583579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:56.583582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:56.583728 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:56.584096 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:56.584103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:56.584346 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 15:16:56.584543 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 15:16:56.584910 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":122,"threads_started":1}
I20260927 15:16:56.587200 14471 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38683
I20260927 15:16:56.587460 14471 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:56.587984 14471 heartbeater.cc:511] Master 127.9.242.190:38683 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:16:56.589473 14281 ts_manager.cc:194] Registered new tserver with Master: c06ebbdd991d4386bf3cbd051b0f8e3d (127.9.242.129:46107)
I20260927 15:16:56.590669 14281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:35371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:16:56.605458 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38683
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:56.712491 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:56.712757 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:56.712805 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:56.716962 14476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:56.717036 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:56.717098 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:56.722590 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:56.723742 14476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:56.725075 14476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:56.734555 14482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:56.734565 14481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:56.734683 14476 server_base.cc:1034] running on GCE node
W20260927 15:16:56.734579 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:56.735183 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:56.735801 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:56.736966 14476 hybrid_clock.cc:648] HybridClock initialized: now 1790522216736908 us; error 70 us; skew 500 ppm
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:56.739775 14476 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:16:56.740528 14476 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:56.740598 14476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:56.740790 14476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:56.742790 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "6ddcfffb975d4cf785125520ad2c0420"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "b972368f8cd2fdff8909a039a0cf6f0f"
server_key_iv: "ad383ebded724307e1b4e0b437463a86"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.743316 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "6ddcfffb975d4cf785125520ad2c0420"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "b972368f8cd2fdff8909a039a0cf6f0f"
server_key_iv: "ad383ebded724307e1b4e0b437463a86"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.746832 14476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:56.749133 14490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.750319 14476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:56.750415 14476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "6ddcfffb975d4cf785125520ad2c0420"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "b972368f8cd2fdff8909a039a0cf6f0f"
server_key_iv: "ad383ebded724307e1b4e0b437463a86"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.750502 14476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:56.765036 14476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:56.768139 14476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:56.768301 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:56.768822 14476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:56.769594 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:56.769639 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.769713 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:56.769739 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.777997 14476 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:42317
I20260927 15:16:56.778015 14603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:42317 every 8 connection(s)
I20260927 15:16:56.778591 14476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:56.780962 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 14476
I20260927 15:16:56.781061 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:56.781338 10186 external_mini_cluster.cc:1428] Setting key 93581ca5a6f8d7d5a3238a138ae54525
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:16:56.792368 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.780249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:46431 (local address 127.9.242.190:38683)
0927 15:16:56.780371 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:56.780375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:56.781278 (+ 903us) server_negotiation.cc:407] Connection header received
0927 15:16:56.782547 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:56.782550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:56.782599 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:56.782703 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:56.784236 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.785041 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.786254 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.786497 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.786583 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:56.789081 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:56.789085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:56.789089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:56.789114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:56.790559 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.791026 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.791029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.791030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.791097 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.791386 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.791388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.791389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.791556 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:56.791626 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:56.792098 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 15:16:56.792242 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:16:56.793251 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.780666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38683 (local address 127.9.242.130:46431)
0927 15:16:56.781122 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:56.781172 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:56.782366 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:56.782877 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:56.782886 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:56.783361 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:56.784076 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:56.784098 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.785170 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.785173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:56.786143 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:56.786152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.786634 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.786638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:56.786797 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:56.787381 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:56.787398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:56.788914 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:56.790667 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:56.790674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:56.790691 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:56.790941 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:56.791179 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:56.791181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:56.791183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:56.791314 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:56.791625 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:56.791629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:56.791874 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 15:16:56.792072 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 15:16:56.792386 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260927 15:16:56.794493 14604 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38683
I20260927 15:16:56.794754 14604 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:56.795192 14604 heartbeater.cc:511] Master 127.9.242.190:38683 requested a full tablet report, sending...
I20260927 15:16:56.796202 14281 ts_manager.cc:194] Registered new tserver with Master: 6ddcfffb975d4cf785125520ad2c0420 (127.9.242.130:42317)
I20260927 15:16:56.796826 14281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:46431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:16:56.806049 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38683
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:16:56.913456 14609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:56.913761 14609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:56.913830 14609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:56.917989 14609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:56.918045 14609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:56.918124 14609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:56.923832 14609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:38683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:56.925185 14609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:56.926496 14609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:56.935542 14614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:56.935515 14617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:56.935534 14615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:56.935585 14609 server_base.cc:1034] running on GCE node
I20260927 15:16:56.936106 14609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:56.936666 14609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:56.937815 14609 hybrid_clock.cc:648] HybridClock initialized: now 1790522216937797 us; error 28 us; skew 500 ppm
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:16:56.940358 14609 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:16:56.941057 14609 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:56.941125 14609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:56.941291 14609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:56.943203 14609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "3a8d7fa4a921448e89ea3f8801cde12a"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "f3c89ce39effa0838dd3305813006f09"
server_key_iv: "120070e61ac933c3d06865f1acb4ef62"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.943676 14609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "3a8d7fa4a921448e89ea3f8801cde12a"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "f3c89ce39effa0838dd3305813006f09"
server_key_iv: "120070e61ac933c3d06865f1acb4ef62"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.947081 14609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:56.949507 14623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.950703 14609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:56.950798 14609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "3a8d7fa4a921448e89ea3f8801cde12a"
format_stamp: "Formatted at 2026-09-27 15:16:56 on dist-test-slave-m4lt"
server_key: "f3c89ce39effa0838dd3305813006f09"
server_key_iv: "120070e61ac933c3d06865f1acb4ef62"
server_key_version: "encryptionkey@0"
I20260927 15:16:56.950904 14609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:56.961699 14609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:56.964331 14609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:56.964493 14609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:56.965049 14609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:56.965874 14609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:56.965921 14609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.965963 14609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:56.965991 14609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:56.974335 14609 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:38537
I20260927 15:16:56.974393 14736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:38537 every 8 connection(s)
I20260927 15:16:56.974853 14609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:56.981441 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 14609
I20260927 15:16:56.981525 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:56.981838 10186 external_mini_cluster.cc:1428] Setting key d9e2b6c9b4d58aa9a7f91a72392a4523
Sep 27 15:16:56 dist-test-slave-m4lt krb5kdc[14237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522216, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:16:56.987037 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.976402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:57297 (local address 127.9.242.190:38683)
0927 15:16:56.976522 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:56.976526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:56.977331 (+ 805us) server_negotiation.cc:407] Connection header received
0927 15:16:56.978621 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:56.978625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:56.978687 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:56.978763 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:56.979846 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.980361 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.981431 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.981761 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.981847 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:56.984305 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:56.984308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:56.984310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:56.984332 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:56.985424 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.985909 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.985911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.985912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.985979 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:56.986214 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:56.986216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:56.986217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:56.986369 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:56.986433 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:56.986836 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 15:16:56.986924 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:16:56.987802 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.976766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:38683 (local address 127.9.242.131:57297)
0927 15:16:56.977193 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:56.977235 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:56.978427 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:56.978925 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:56.978931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:56.979267 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:56.979716 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:56.979728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.980468 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.980471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:56.981321 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:56.981330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.981900 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.981904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:56.982055 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:56.982609 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:16:56.982629 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:56.984168 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:56.985519 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:56.985524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:56.985534 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:56.985839 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:56.986041 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:16:56.986043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:16:56.986044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:16:56.986156 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:16:56.986432 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:56.986436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:56.986637 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 15:16:56.986806 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 15:16:56.987028 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
I20260927 15:16:56.988731 14737 heartbeater.cc:347] Connected to a master server at 127.9.242.190:38683
I20260927 15:16:56.988914 14737 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:56.989285 14737 heartbeater.cc:511] Master 127.9.242.190:38683 requested a full tablet report, sending...
I20260927 15:16:56.990134 14281 ts_manager.cc:194] Registered new tserver with Master: 3a8d7fa4a921448e89ea3f8801cde12a (127.9.242.131:38537)
I20260927 15:16:56.990638 14281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:57297
I20260927 15:16:56.995177 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:57.003594 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:56.996985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.9.242.190:38683)
0927 15:16:56.997136 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:56.997140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:56.997310 (+ 170us) server_negotiation.cc:407] Connection header received
0927 15:16:56.997394 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:56.997396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:56.997437 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:56.997501 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:56.998175 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.998676 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.999411 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:56.999625 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:56.999704 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.000940 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:57.000944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:57.000946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:16:57.000967 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:57.002143 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:57.002489 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:57.002492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:57.002493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:57.002559 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:16:57.002767 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:16:57.002769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:16:57.002770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:16:57.002967 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:57.003060 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.003371 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.003465 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 15:16:57.006428 14281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:57.009356 14281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:57.024057 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.018136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:46107 (local address 127.0.0.1:60114)
0927 15:16:57.018694 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.018739 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:57.018877 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:57.019430 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:57.019437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:57.019462 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:57.019612 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.019619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.020985 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.020988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.022410 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.022420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.023523 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.023527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.023630 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.023673 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:16:57.023753 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:16:57.023870 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":126,"threads_started":1}
I20260927 15:16:57.024467 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.018896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:42317 (local address 127.0.0.1:36962)
0927 15:16:57.019678 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.019692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:57.019784 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:57.020315 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:57.020318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:57.020333 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:57.020430 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.020437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.021943 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.021946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.023101 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.023111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.024113 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.024116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.024200 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.024220 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:16:57.024272 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 15:16:57.024330 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":89,"threads_started":1}
I20260927 15:16:57.024467 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.019191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:38537 (local address 127.0.0.1:44548)
0927 15:16:57.019944 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.019955 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:57.020024 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:57.020550 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:57.020552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:57.020565 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:57.020628 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.020632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.021639 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.021642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.022677 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.022693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.023913 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.023915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.023988 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.024018 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:16:57.024082 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:16:57.024148 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":83,"threads_started":1}
I20260927 15:16:57.024688 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.018281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60114 (local address 127.9.242.129:46107)
0927 15:16:57.018515 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.018522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.018820 (+ 298us) server_negotiation.cc:407] Connection header received
0927 15:16:57.019044 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.019050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.019275 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.019440 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:57.019715 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.020846 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.022552 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.023402 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.024261 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.024397 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.024448 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.024539 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260927 15:16:57.025202 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.019566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.9.242.130:42317)
0927 15:16:57.019891 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.019904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.019927 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:16:57.019995 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.020000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.020180 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.020318 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:57.020513 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.021806 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.023242 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.023991 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.024104 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.024487 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.024699 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.025066 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":136,"threads_started":1}
I20260927 15:16:57.025635 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.019334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.9.242.131:38537)
0927 15:16:57.020004 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.020016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.020038 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:16:57.020112 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.020117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.020269 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.020407 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:57.020726 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.021507 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.022818 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.023645 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.023745 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.023864 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.024528 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.024634 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":143,"threads_started":1}
I20260927 15:16:57.027829 14405 tablet_service.cc:1467] Processing CreateTablet for tablet dabbba6e98df4e6dae249cdc406787b4 (DEFAULT_TABLE table=test-table [id=af4d1c50adb54cb68ddea2dd3bca068f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:57.028030 14538 tablet_service.cc:1467] Processing CreateTablet for tablet dabbba6e98df4e6dae249cdc406787b4 (DEFAULT_TABLE table=test-table [id=af4d1c50adb54cb68ddea2dd3bca068f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:57.028149 14671 tablet_service.cc:1467] Processing CreateTablet for tablet dabbba6e98df4e6dae249cdc406787b4 (DEFAULT_TABLE table=test-table [id=af4d1c50adb54cb68ddea2dd3bca068f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:57.028945 14405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabbba6e98df4e6dae249cdc406787b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:57.028945 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabbba6e98df4e6dae249cdc406787b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:57.029021 14538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabbba6e98df4e6dae249cdc406787b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:57.034937 14754 tablet_bootstrap.cc:492] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Bootstrap starting.
I20260927 15:16:57.036464 14755 tablet_bootstrap.cc:492] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Bootstrap starting.
I20260927 15:16:57.037598 14754 tablet_bootstrap.cc:654] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:57.037958 14756 tablet_bootstrap.cc:492] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: Bootstrap starting.
I20260927 15:16:57.038542 14755 tablet_bootstrap.cc:654] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:57.038928 14754 log.cc:824] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:57.039665 14755 log.cc:824] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:57.039855 14756 tablet_bootstrap.cc:654] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:57.040931 14756 log.cc:824] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:57.041033 14754 tablet_bootstrap.cc:492] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: No bootstrap required, opened a new log
I20260927 15:16:57.041213 14754 ts_tablet_manager.cc:1397] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 15:16:57.041226 14755 tablet_bootstrap.cc:492] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: No bootstrap required, opened a new log
I20260927 15:16:57.041394 14755 ts_tablet_manager.cc:1397] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:16:57.042841 14756 tablet_bootstrap.cc:492] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: No bootstrap required, opened a new log
I20260927 15:16:57.043015 14756 ts_tablet_manager.cc:1397] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:57.044289 14755 raft_consensus.cc:348] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.044557 14755 raft_consensus.cc:374] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:57.044615 14755 raft_consensus.cc:729] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8d7fa4a921448e89ea3f8801cde12a, State: Initialized, Role: FOLLOWER
I20260927 15:16:57.044953 14754 raft_consensus.cc:348] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.045082 14755 consensus_queue.cc:260] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.045202 14754 raft_consensus.cc:374] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:57.045259 14754 raft_consensus.cc:729] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ddcfffb975d4cf785125520ad2c0420, State: Initialized, Role: FOLLOWER
I20260927 15:16:57.045701 14754 consensus_queue.cc:260] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.046080 14755 ts_tablet_manager.cc:1428] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:16:57.046602 14754 ts_tablet_manager.cc:1428] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:16:57.046749 14737 heartbeater.cc:503] Master 127.9.242.190:38683 was elected leader, sending a full tablet report...
I20260927 15:16:57.046646 14756 raft_consensus.cc:348] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.046856 14756 raft_consensus.cc:374] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:57.046926 14756 raft_consensus.cc:729] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06ebbdd991d4386bf3cbd051b0f8e3d, State: Initialized, Role: FOLLOWER
I20260927 15:16:57.047106 14604 heartbeater.cc:503] Master 127.9.242.190:38683 was elected leader, sending a full tablet report...
I20260927 15:16:57.047294 14756 consensus_queue.cc:260] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.047838 14471 heartbeater.cc:503] Master 127.9.242.190:38683 was elected leader, sending a full tablet report...
I20260927 15:16:57.049233 14756 ts_tablet_manager.cc:1428] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 15:16:57.072469 14472 tablet.cc:2369] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:57.225708 14738 tablet.cc:2369] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:57.280107 14605 tablet.cc:2369] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:57.443660 14762 raft_consensus.cc:482] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:57.443892 14762 raft_consensus.cc:504] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.445091 14762 leader_election.cc:288] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ddcfffb975d4cf785125520ad2c0420 (127.9.242.130:42317), 3a8d7fa4a921448e89ea3f8801cde12a (127.9.242.131:38537)
I20260927 15:16:57.449244 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.445488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:34141 (local address 127.9.242.130:42317)
0927 15:16:57.445681 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.445687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.445786 (+ 99us) server_negotiation.cc:407] Connection header received
0927 15:16:57.445942 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.445945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.445997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.446074 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:57.446487 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.447083 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.448308 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.448718 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.448807 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.448862 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.449047 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.449116 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:57.449276 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.445419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:42317 (local address 127.9.242.129:34141)
0927 15:16:57.445688 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.445704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:57.445841 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:57.446113 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:57.446116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:57.446239 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:57.446370 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.446377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.447190 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.447194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.448184 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.448194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.448828 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.448831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.448924 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.448947 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:16:57.449012 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:16:57.449079 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":46,"spinlock_wait_cycles":176384,"thread_start_us":79,"threads_started":1}
I20260927 15:16:57.449338 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.445928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:44803 (local address 127.9.242.131:38537)
0927 15:16:57.446047 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.446051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.446104 (+ 53us) server_negotiation.cc:407] Connection header received
0927 15:16:57.446375 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.446378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.446428 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.446501 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:16:57.446746 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.447285 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.448357 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.448748 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.448862 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.448911 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.449158 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.449222 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:57.449383 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.445799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:38537 (local address 127.9.242.129:44803)
0927 15:16:57.446043 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.446055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:57.446244 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:57.446516 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:57.446519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:57.446535 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:16:57.446645 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.446651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.447385 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.447388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.448240 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:57.448251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.448868 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.448872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:57.448997 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.449018 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:16:57.449122 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:16:57.449199 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":1,"spinlock_wait_cycles":37120,"thread_start_us":87,"threads_started":1}
I20260927 15:16:57.449945 14691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabbba6e98df4e6dae249cdc406787b4" candidate_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" is_pre_election: true
I20260927 15:16:57.449950 14558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabbba6e98df4e6dae249cdc406787b4" candidate_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ddcfffb975d4cf785125520ad2c0420" is_pre_election: true
I20260927 15:16:57.450214 14691 raft_consensus.cc:2454] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06ebbdd991d4386bf3cbd051b0f8e3d in term 0.
I20260927 15:16:57.450220 14558 raft_consensus.cc:2454] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06ebbdd991d4386bf3cbd051b0f8e3d in term 0.
I20260927 15:16:57.450634 14358 leader_election.cc:302] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ddcfffb975d4cf785125520ad2c0420, c06ebbdd991d4386bf3cbd051b0f8e3d; no voters:
I20260927 15:16:57.450850 14762 raft_consensus.cc:2781] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:57.450924 14762 raft_consensus.cc:482] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:57.450973 14762 raft_consensus.cc:3037] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:57.451963 14762 raft_consensus.cc:504] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.452270 14762 leader_election.cc:288] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [CANDIDATE]: Term 1 election: Requested vote from peers 6ddcfffb975d4cf785125520ad2c0420 (127.9.242.130:42317), 3a8d7fa4a921448e89ea3f8801cde12a (127.9.242.131:38537)
I20260927 15:16:57.452592 14558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabbba6e98df4e6dae249cdc406787b4" candidate_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ddcfffb975d4cf785125520ad2c0420"
I20260927 15:16:57.452649 14691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabbba6e98df4e6dae249cdc406787b4" candidate_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8d7fa4a921448e89ea3f8801cde12a"
I20260927 15:16:57.452744 14691 raft_consensus.cc:3037] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:57.452755 14558 raft_consensus.cc:3037] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:57.453881 14691 raft_consensus.cc:2454] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06ebbdd991d4386bf3cbd051b0f8e3d in term 1.
I20260927 15:16:57.453933 14558 raft_consensus.cc:2454] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06ebbdd991d4386bf3cbd051b0f8e3d in term 1.
I20260927 15:16:57.454161 14358 leader_election.cc:302] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8d7fa4a921448e89ea3f8801cde12a, c06ebbdd991d4386bf3cbd051b0f8e3d; no voters:
I20260927 15:16:57.454336 14762 raft_consensus.cc:2781] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:57.454536 14762 raft_consensus.cc:686] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [term 1 LEADER]: Becoming Leader. State: Replica: c06ebbdd991d4386bf3cbd051b0f8e3d, State: Running, Role: LEADER
I20260927 15:16:57.454873 14762 consensus_queue.cc:237] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } }
I20260927 15:16:57.457839 14279 catalog_manager.cc:5495] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d reported cstate change: term changed from 0 to 1, leader changed from <none> to c06ebbdd991d4386bf3cbd051b0f8e3d (127.9.242.129). New cstate: current_term: 1 leader_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c06ebbdd991d4386bf3cbd051b0f8e3d" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 } health_report { overall_health: UNKNOWN } } }
I20260927 15:16:57.479900 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.476366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.9.242.129:46107)
0927 15:16:57.476537 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.476542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.476557 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:16:57.476669 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.476672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.476731 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.476825 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:16:57.477085 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.477600 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.478501 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:57.478721 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:57.478808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:57.479108 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:16:57.479527 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:16:57.479646 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.479699 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.479757 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:16:57.486399 14691 raft_consensus.cc:1261] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 1 FOLLOWER]: Refusing update from remote peer c06ebbdd991d4386bf3cbd051b0f8e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:57.486496 14558 raft_consensus.cc:1261] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 1 FOLLOWER]: Refusing update from remote peer c06ebbdd991d4386bf3cbd051b0f8e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:57.487140 14765 consensus_queue.cc:1035] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8d7fa4a921448e89ea3f8801cde12a" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:57.487339 14762 consensus_queue.cc:1035] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ddcfffb975d4cf785125520ad2c0420" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:57.494642 14769 mvcc.cc:205] Tried to move back new op lower bound from 7333979002815954944 to 7333979002697469952. Current Snapshot: MvccSnapshot[applied={T|T < 7333979002815954944}]
I20260927 15:16:57.496151 14770 mvcc.cc:205] Tried to move back new op lower bound from 7333979002815954944 to 7333979002697469952. Current Snapshot: MvccSnapshot[applied={T|T < 7333979002815954944}]
I20260927 15:16:57.502775 14279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:57.502931 14279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:57.505156 14279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b3ae4e8f66847c2ad34857729fe04b1: Sending DeleteTablet for 3 replicas of tablet dabbba6e98df4e6dae249cdc406787b4
I20260927 15:16:57.505959 14538 tablet_service.cc:1514] Processing DeleteTablet for tablet dabbba6e98df4e6dae249cdc406787b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:57 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:36962
I20260927 15:16:57.506040 14671 tablet_service.cc:1514] Processing DeleteTablet for tablet dabbba6e98df4e6dae249cdc406787b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:57 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:44548
I20260927 15:16:57.506071 14405 tablet_service.cc:1514] Processing DeleteTablet for tablet dabbba6e98df4e6dae249cdc406787b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:57 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:60114
I20260927 15:16:57.506524 14779 tablet_replica.cc:330] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: stopping tablet replica
I20260927 15:16:57.506716 14778 tablet_replica.cc:330] T dabbba6e98df4e6dae249cdc406787b4 P c06ebbdd991d4386bf3cbd051b0f8e3d: stopping tablet replica
I20260927 15:16:57.506846 14779 raft_consensus.cc:2229] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:57.507007 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 14343
I20260927 15:16:57.507571 14779 raft_consensus.cc:2258] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:57.508628 14777 tablet_replica.cc:330] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: stopping tablet replica
I20260927 15:16:57.508983 14779 ts_tablet_manager.cc:1905] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:57.509014 14777 raft_consensus.cc:2229] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:57.509289 14777 raft_consensus.cc:2258] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:57.510407 14777 ts_tablet_manager.cc:1905] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:57.511686 14779 ts_tablet_manager.cc:1918] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:57.511783 14779 log.cc:1196] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/dabbba6e98df4e6dae249cdc406787b4
I20260927 15:16:57.512068 14779 ts_tablet_manager.cc:1939] T dabbba6e98df4e6dae249cdc406787b4 P 3a8d7fa4a921448e89ea3f8801cde12a: Deleting consensus metadata
I20260927 15:16:57.512493 14777 ts_tablet_manager.cc:1918] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:57.512596 14777 log.cc:1196] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/dabbba6e98df4e6dae249cdc406787b4
I20260927 15:16:57.512859 14777 ts_tablet_manager.cc:1939] T dabbba6e98df4e6dae249cdc406787b4 P 6ddcfffb975d4cf785125520ad2c0420: Deleting consensus metadata
W20260927 15:16:57.513741 14265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:46107 (error 108)
I20260927 15:16:57.513952 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 14476
I20260927 15:16:57.514183 14265 catalog_manager.cc:4841] TS 3a8d7fa4a921448e89ea3f8801cde12a (127.9.242.131:38537): tablet dabbba6e98df4e6dae249cdc406787b4 (table test-table [id=af4d1c50adb54cb68ddea2dd3bca068f]) successfully deleted
I20260927 15:16:57.515245 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.514976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:46107 (local address 127.0.0.1:60130)
0927 15:16:57.515092 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.515171 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:46107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:16:57.515377 14265 catalog_manager.cc:4584] TS c06ebbdd991d4386bf3cbd051b0f8e3d (127.9.242.129:46107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dabbba6e98df4e6dae249cdc406787b4: Network error: Client connection negotiation failed: client connection to 127.9.242.129:46107: connect: Connection refused (error 111)
I20260927 15:16:57.519286 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 14609
I20260927 15:16:57.520737 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.519560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:42317 (local address 127.0.0.1:36976)
0927 15:16:57.520603 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.520663 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.130:42317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":958}
I20260927 15:16:57.523715 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 14250
2026-09-27T15:16:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2477 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T15:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:57Z Disabled control of system clock
I20260927 15:16:57.551201 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34211
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:16:57.658471 14784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:57.658731 14784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:57.658768 14784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:57.662722 14784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:57.662778 14784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:57.662792 14784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:57.662807 14784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:57.662822 14784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:16:57.667516 14784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:34211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:57.668540 14784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:57.669997 14784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:16:57.678366 14784 server_base.cc:1034] running on GCE node
W20260927 15:16:57.678318 14792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:57.678315 14789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:57.678314 14790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:57.679261 14784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:57.680379 14784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:57.681553 14784 hybrid_clock.cc:648] HybridClock initialized: now 1790522217681523 us; error 47 us; skew 500 ppm
I20260927 15:16:57.683559 14784 webserver.cc:460] Webserver started at http://127.9.242.190:43819/ using document root <none> and password file <none>
I20260927 15:16:57.684140 14784 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:57.684211 14784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:57.684412 14784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:57.686499 14784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "4eb225211245403ebece2a6b2f934e13"
format_stamp: "Formatted at 2026-09-27 15:16:57 on dist-test-slave-m4lt"
server_key: "a22317e658c12ee98bab23755dbc539c"
server_key_iv: "ea781b06f260d899fc7811f865d65e93"
server_key_version: "encryptionkey@0"
I20260927 15:16:57.687027 14784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "4eb225211245403ebece2a6b2f934e13"
format_stamp: "Formatted at 2026-09-27 15:16:57 on dist-test-slave-m4lt"
server_key: "a22317e658c12ee98bab23755dbc539c"
server_key_iv: "ea781b06f260d899fc7811f865d65e93"
server_key_version: "encryptionkey@0"
I20260927 15:16:57.690662 14784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:57.692977 14798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:57.694304 14784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:57.694409 14784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "4eb225211245403ebece2a6b2f934e13"
format_stamp: "Formatted at 2026-09-27 15:16:57 on dist-test-slave-m4lt"
server_key: "a22317e658c12ee98bab23755dbc539c"
server_key_iv: "ea781b06f260d899fc7811f865d65e93"
server_key_version: "encryptionkey@0"
I20260927 15:16:57.694501 14784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:57.698863 14784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:57.699436 14784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:57.699585 14784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:57.706055 14784 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:34211
I20260927 15:16:57.706099 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:34211 every 8 connection(s)
I20260927 15:16:57.706985 14784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:57.710139 14851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:57.715525 14851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13: Bootstrap starting.
I20260927 15:16:57.716389 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 14784
I20260927 15:16:57.716503 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:57.716792 10186 external_mini_cluster.cc:1428] Setting key 88093dcc72eb04c3a181095f779679b6
I20260927 15:16:57.718554 14851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:57.719385 14851 log.cc:824] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:57.721717 14851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13: No bootstrap required, opened a new log
I20260927 15:16:57.722513 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.718036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.9.242.190:34211)
0927 15:16:57.718697 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.718713 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.718752 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:16:57.719413 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.719431 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.719851 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.720233 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:57.721079 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:57.721084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:57.721095 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:57.721105 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:57.721167 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:57.721183 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:57.721268 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.721390 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.721628 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260927 15:16:57.725286 14851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb225211245403ebece2a6b2f934e13" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34211 } }
I20260927 15:16:57.725505 14851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:57.725539 14851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb225211245403ebece2a6b2f934e13, State: Initialized, Role: FOLLOWER
I20260927 15:16:57.725999 14851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb225211245403ebece2a6b2f934e13" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34211 } }
I20260927 15:16:57.726136 14851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:57.726197 14851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:57.726292 14851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:57.727238 14851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb225211245403ebece2a6b2f934e13" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34211 } }
I20260927 15:16:57.727550 14851 leader_election.cc:302] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eb225211245403ebece2a6b2f934e13; no voters:
I20260927 15:16:57.727851 14851 leader_election.cc:288] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:57.727942 14856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:57.728184 14856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb225211245403ebece2a6b2f934e13, State: Running, Role: LEADER
I20260927 15:16:57.728462 14856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb225211245403ebece2a6b2f934e13" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34211 } }
I20260927 15:16:57.728995 14851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:57.730048 14857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eb225211245403ebece2a6b2f934e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb225211245403ebece2a6b2f934e13" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34211 } } }
I20260927 15:16:57.730245 14858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eb225211245403ebece2a6b2f934e13. Latest consensus state: current_term: 1 leader_uuid: "4eb225211245403ebece2a6b2f934e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eb225211245403ebece2a6b2f934e13" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34211 } } }
I20260927 15:16:57.730348 14858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:57.730288 14857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:57.731518 14865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:57.734211 14865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:57.739239 14865 catalog_manager.cc:1347] Generated new cluster ID: f4abfa505cce49a9bb60f512566f782c
I20260927 15:16:57.739311 14865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:57.758476 14865 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:57.758801 14865 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:57.763351 14865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13: Generated new TSK 0
I20260927 15:16:57.764195 14865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:16:57.769924 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34211
--builtin_ntp_servers=127.9.242.148:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 15:16:57.878194 14875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:57.878435 14875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:57.878471 14875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:57.882400 14875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:57.882447 14875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:57.882508 14875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:16:57.887244 14875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:57.888283 14875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:57.889605 14875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:57.898692 14881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:57.898692 14883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:57.898818 14875 server_base.cc:1034] running on GCE node
W20260927 15:16:57.898968 14880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:57.899426 14875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:57.900058 14875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:57.901239 14875 hybrid_clock.cc:648] HybridClock initialized: now 1790522217901219 us; error 33 us; skew 500 ppm
I20260927 15:16:57.903117 14875 webserver.cc:460] Webserver started at http://127.9.242.129:46653/ using document root <none> and password file <none>
I20260927 15:16:57.903668 14875 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:57.903734 14875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:57.903913 14875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:57.906010 14875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "dae21df6deb645bebd558b6a1a638b40"
format_stamp: "Formatted at 2026-09-27 15:16:57 on dist-test-slave-m4lt"
server_key: "469a7b4dfd757d8052692dd9e1a8c01d"
server_key_iv: "cb90078117c83d7171d55d5df5ce6232"
server_key_version: "encryptionkey@0"
I20260927 15:16:57.906531 14875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "dae21df6deb645bebd558b6a1a638b40"
format_stamp: "Formatted at 2026-09-27 15:16:57 on dist-test-slave-m4lt"
server_key: "469a7b4dfd757d8052692dd9e1a8c01d"
server_key_iv: "cb90078117c83d7171d55d5df5ce6232"
server_key_version: "encryptionkey@0"
I20260927 15:16:57.910136 14875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:16:57.912547 14889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:57.913873 14875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:57.913971 14875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "dae21df6deb645bebd558b6a1a638b40"
format_stamp: "Formatted at 2026-09-27 15:16:57 on dist-test-slave-m4lt"
server_key: "469a7b4dfd757d8052692dd9e1a8c01d"
server_key_iv: "cb90078117c83d7171d55d5df5ce6232"
server_key_version: "encryptionkey@0"
I20260927 15:16:57.914062 14875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:57.918460 14875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:57.919019 14875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:57.919167 14875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:57.919667 14875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:57.920485 14875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:57.920528 14875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:57.920575 14875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:57.920610 14875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:57.929901 14875 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:34235
I20260927 15:16:57.929945 15002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:34235 every 8 connection(s)
I20260927 15:16:57.930858 14875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:16:57.935491 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 14875
I20260927 15:16:57.935626 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:16:57.935911 10186 external_mini_cluster.cc:1428] Setting key 6cb05167d75f57aa784307f3cb82ea37
I20260927 15:16:57.937879 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34211
--builtin_ntp_servers=127.9.242.148:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:16:57.939618 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.932751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:40865 (local address 127.9.242.190:34211)
0927 15:16:57.932879 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:57.932882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:57.934176 (+ 1294us) server_negotiation.cc:407] Connection header received
0927 15:16:57.935576 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:57.935579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:57.935634 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:57.935700 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:57.937049 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:57.937053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:57.937056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:57.937068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:57.937092 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:57.937110 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:57.937185 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:57.939427 (+ 2242us) server_negotiation.cc:299] Negotiation successful
0927 15:16:57.939474 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:16:57.940712 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:57.933292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34211 (local address 127.9.242.129:40865)
0927 15:16:57.934033 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:57.934073 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:57.935364 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:57.935896 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:57.935903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:57.936009 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:57.936739 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:57.936742 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:57.936799 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:57.936890 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:57.937176 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:57.937184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:57.937189 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:16:57.939452 (+ 2263us) client_negotiation.cc:240] Negotiation successful
0927 15:16:57.939723 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":132,"threads_started":1}
I20260927 15:16:57.942030 15003 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34211
I20260927 15:16:57.942283 15003 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:57.942741 15003 heartbeater.cc:511] Master 127.9.242.190:34211 requested a full tablet report, sending...
I20260927 15:16:57.944499 14815 ts_manager.cc:194] Registered new tserver with Master: dae21df6deb645bebd558b6a1a638b40 (127.9.242.129:34235)
W20260927 15:16:58.058027 15006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:58.058290 15006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:58.058328 15006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:58.062357 15006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:58.062415 15006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:58.062479 15006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:16:58.067495 15006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:58.068645 15006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:58.070071 15006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:58.079885 15014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:58.079849 15011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:58.079831 15006 server_base.cc:1034] running on GCE node
W20260927 15:16:58.079772 15012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:58.080464 15006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:58.081123 15006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:58.082316 15006 hybrid_clock.cc:648] HybridClock initialized: now 1790522218082288 us; error 40 us; skew 500 ppm
I20260927 15:16:58.084280 15006 webserver.cc:460] Webserver started at http://127.9.242.130:35277/ using document root <none> and password file <none>
I20260927 15:16:58.084846 15006 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:58.084903 15006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:58.085083 15006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:58.087378 15006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "695c1e47a7e949c58afa0b2e7d353106"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "4f98ecd05f0f06d85cb9767e75b3f805"
server_key_iv: "4a35a94ff5fb02ebcb3ce750ae061466"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.087937 15006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "695c1e47a7e949c58afa0b2e7d353106"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "4f98ecd05f0f06d85cb9767e75b3f805"
server_key_iv: "4a35a94ff5fb02ebcb3ce750ae061466"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.091677 15006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:16:58.094172 15020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:58.095479 15006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:58.095589 15006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "695c1e47a7e949c58afa0b2e7d353106"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "4f98ecd05f0f06d85cb9767e75b3f805"
server_key_iv: "4a35a94ff5fb02ebcb3ce750ae061466"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.095697 15006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:58.100509 15006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:58.101147 15006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:58.101302 15006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:58.101861 15006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:58.102728 15006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:58.102772 15006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:58.102809 15006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:58.102834 15006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:58.111158 15006 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:38889
I20260927 15:16:58.111240 15133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:38889 every 8 connection(s)
I20260927 15:16:58.112031 15006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:16:58.114224 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15006
I20260927 15:16:58.114343 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:16:58.114640 10186 external_mini_cluster.cc:1428] Setting key 65b2c6fa75252cf276935c545f99d22f
I20260927 15:16:58.116581 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34211
--builtin_ntp_servers=127.9.242.148:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 15:16:58.118520 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.113740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:35161 (local address 127.9.242.190:34211)
0927 15:16:58.113888 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.113892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.115073 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 15:16:58.116650 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.116653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.116712 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.116783 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.118036 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.118039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.118041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.118052 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.118071 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.118078 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.118165 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.118365 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.118399 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:16:58.119608 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.114227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34211 (local address 127.9.242.130:35161)
0927 15:16:58.114920 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.114961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.116450 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.116954 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.116962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.117066 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:58.117595 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:58.117597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:58.117695 (+ 98us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:58.117798 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:58.118190 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:58.118198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:58.118205 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 15:16:58.118437 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 15:16:58.118807 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
I20260927 15:16:58.120688 15134 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34211
I20260927 15:16:58.120896 15134 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:58.121337 15134 heartbeater.cc:511] Master 127.9.242.190:34211 requested a full tablet report, sending...
I20260927 15:16:58.122393 14815 ts_manager.cc:194] Registered new tserver with Master: 695c1e47a7e949c58afa0b2e7d353106 (127.9.242.130:38889)
W20260927 15:16:58.226468 15137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:58.226753 15137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:58.226794 15137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:58.230923 15137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:58.230983 15137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:58.231042 15137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:16:58.235860 15137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:58.236860 15137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:58.238221 15137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:58.247479 15143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:58.247471 15142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:58.247471 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:58.247715 15137 server_base.cc:1034] running on GCE node
I20260927 15:16:58.248096 15137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:58.248708 15137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:58.249858 15137 hybrid_clock.cc:648] HybridClock initialized: now 1790522218249828 us; error 41 us; skew 500 ppm
I20260927 15:16:58.251636 15137 webserver.cc:460] Webserver started at http://127.9.242.131:41529/ using document root <none> and password file <none>
I20260927 15:16:58.252179 15137 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:58.252238 15137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:58.252420 15137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:58.254438 15137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "efec3528e0054636afd07867513b63ff"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "6de1aa95c889d1830828974416eafc9c"
server_key_iv: "8e0ae7a0e3d639177729e9b6efb94993"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.254951 15137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "efec3528e0054636afd07867513b63ff"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "6de1aa95c889d1830828974416eafc9c"
server_key_iv: "8e0ae7a0e3d639177729e9b6efb94993"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.258529 15137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:58.261006 15151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:58.262311 15137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:58.262410 15137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "efec3528e0054636afd07867513b63ff"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "6de1aa95c889d1830828974416eafc9c"
server_key_iv: "8e0ae7a0e3d639177729e9b6efb94993"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.262506 15137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:58.266952 15137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:58.267540 15137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:58.267689 15137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:58.268191 15137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:16:58.269019 15137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:16:58.269064 15137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:58.269112 15137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:16:58.269145 15137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:58.276852 15137 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:45885
I20260927 15:16:58.276902 15264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:45885 every 8 connection(s)
I20260927 15:16:58.277781 15137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:16:58.283717 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.279593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:34049 (local address 127.9.242.190:34211)
0927 15:16:58.279732 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.279736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.280762 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 15:16:58.282108 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.282112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.282158 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.282231 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.283339 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.283342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.283344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.283353 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.283372 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.283379 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.283450 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.283536 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.283569 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:16:58.283885 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15137
I20260927 15:16:58.284016 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:16:58.284298 10186 external_mini_cluster.cc:1428] Setting key 47cb80bfe2a3fba92202bd6e3cc0d6b6
I20260927 15:16:58.284770 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.280016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34211 (local address 127.9.242.131:34049)
0927 15:16:58.280630 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.280677 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.281907 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.282425 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.282432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.282522 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:58.283049 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:58.283051 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:58.283091 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:58.283215 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:58.283440 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:58.283445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:58.283448 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:16:58.283567 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 15:16:58.283786 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
I20260927 15:16:58.285796 15265 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34211
I20260927 15:16:58.286005 15265 heartbeater.cc:464] Registering TS with master...
I20260927 15:16:58.286382 15265 heartbeater.cc:511] Master 127.9.242.190:34211 requested a full tablet report, sending...
I20260927 15:16:58.287385 14815 ts_manager.cc:194] Registered new tserver with Master: efec3528e0054636afd07867513b63ff (127.9.242.131:45885)
I20260927 15:16:58.297859 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:16:58.301290 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.299799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.9.242.190:34211)
0927 15:16:58.299917 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.299922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.300129 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:16:58.300236 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.300239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.300274 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.300337 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.300940 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.300944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.300947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.300958 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.300977 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.300985 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.301051 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.301127 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.301166 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:16:58.303743 14815 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:16:58.306741 14815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:16:58.317263 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.314551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:45885 (local address 127.0.0.1:51520)
0927 15:16:58.315347 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.315395 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.315483 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.316013 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.316018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.316091 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:58.316592 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:58.316604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:58.316624 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:58.316680 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:58.316958 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:58.316966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:58.316971 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:16:58.317066 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:16:58.317129 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":104,"threads_started":1}
I20260927 15:16:58.317276 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.315225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.9.242.131:45885)
0927 15:16:58.315601 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.315613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.315636 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:16:58.315689 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.315694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.315888 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.316008 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.316791 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.316797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.316809 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.316821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.316860 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.316892 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.316975 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.317078 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.317153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":128,"threads_started":1}
I20260927 15:16:58.318030 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.314788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:38889 (local address 127.0.0.1:56680)
0927 15:16:58.315695 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.315709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.315785 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.316789 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.316792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.316806 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:58.317209 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:58.317211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:58.317222 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:58.317248 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:58.317784 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:58.317786 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:58.317789 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:16:58.317876 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:16:58.317908 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":77,"threads_started":1}
I20260927 15:16:58.318521 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.314610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.9.242.129:34235)
0927 15:16:58.314994 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.315004 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.315810 (+ 806us) server_negotiation.cc:407] Connection header received
0927 15:16:58.315893 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.315898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.316072 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.316242 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.317504 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.317508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.317515 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.317523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.317559 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.317582 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.317700 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.318298 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.318368 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":87,"threads_started":1}
I20260927 15:16:58.318621 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.315225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.9.242.130:38889)
0927 15:16:58.315660 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.315669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.315865 (+ 196us) server_negotiation.cc:407] Connection header received
0927 15:16:58.315912 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.315916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.316072 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.316190 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.317495 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.317501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.317513 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.317523 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.317559 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.317608 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.317755 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.317913 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.318474 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":100,"threads_started":1}
I20260927 15:16:58.318720 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.314475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:34235 (local address 127.0.0.1:51774)
0927 15:16:58.315718 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.315731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.315809 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.316352 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.316355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.316368 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:58.316806 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:58.316808 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:58.316817 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:58.316836 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:58.317739 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:58.317742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:58.317745 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:16:58.318105 (+ 360us) client_negotiation.cc:240] Negotiation successful
0927 15:16:58.318150 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":74,"threads_started":1}
I20260927 15:16:58.320333 15199 tablet_service.cc:1467] Processing CreateTablet for tablet b19679c05a8e4ffaa099155472ddc512 (DEFAULT_TABLE table=test-table [id=b6ea107770c944e99dca561771b171f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:58.320964 15068 tablet_service.cc:1467] Processing CreateTablet for tablet b19679c05a8e4ffaa099155472ddc512 (DEFAULT_TABLE table=test-table [id=b6ea107770c944e99dca561771b171f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:58.321470 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19679c05a8e4ffaa099155472ddc512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:58.321331 14937 tablet_service.cc:1467] Processing CreateTablet for tablet b19679c05a8e4ffaa099155472ddc512 (DEFAULT_TABLE table=test-table [id=b6ea107770c944e99dca561771b171f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:16:58.321919 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19679c05a8e4ffaa099155472ddc512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:58.322341 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19679c05a8e4ffaa099155472ddc512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:58.327786 15282 tablet_bootstrap.cc:492] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Bootstrap starting.
I20260927 15:16:58.329210 15284 tablet_bootstrap.cc:492] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Bootstrap starting.
I20260927 15:16:58.330317 15282 tablet_bootstrap.cc:654] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:58.331044 15284 tablet_bootstrap.cc:654] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:58.331341 15282 log.cc:824] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:58.331764 15284 log.cc:824] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:58.331949 15283 tablet_bootstrap.cc:492] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40: Bootstrap starting.
I20260927 15:16:58.333284 15284 tablet_bootstrap.cc:492] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: No bootstrap required, opened a new log
I20260927 15:16:58.333346 15282 tablet_bootstrap.cc:492] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: No bootstrap required, opened a new log
I20260927 15:16:58.333460 15284 ts_tablet_manager.cc:1397] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:16:58.333518 15282 ts_tablet_manager.cc:1397] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:16:58.334026 15283 tablet_bootstrap.cc:654] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:58.334798 15283 log.cc:824] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:58.336760 15283 tablet_bootstrap.cc:492] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40: No bootstrap required, opened a new log
I20260927 15:16:58.336938 15283 ts_tablet_manager.cc:1397] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:16:58.337220 15284 raft_consensus.cc:348] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.337458 15284 raft_consensus.cc:374] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:58.337510 15284 raft_consensus.cc:729] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efec3528e0054636afd07867513b63ff, State: Initialized, Role: FOLLOWER
I20260927 15:16:58.337930 15284 consensus_queue.cc:260] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.337890 15282 raft_consensus.cc:348] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.338128 15282 raft_consensus.cc:374] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:58.338177 15282 raft_consensus.cc:729] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695c1e47a7e949c58afa0b2e7d353106, State: Initialized, Role: FOLLOWER
I20260927 15:16:58.338587 15282 consensus_queue.cc:260] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.338776 15284 ts_tablet_manager.cc:1428] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:16:58.339141 15265 heartbeater.cc:503] Master 127.9.242.190:34211 was elected leader, sending a full tablet report...
I20260927 15:16:58.339186 15134 heartbeater.cc:503] Master 127.9.242.190:34211 was elected leader, sending a full tablet report...
I20260927 15:16:58.339506 15282 ts_tablet_manager.cc:1428] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 15:16:58.341939 15283 raft_consensus.cc:348] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.342193 15283 raft_consensus.cc:374] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:58.342259 15283 raft_consensus.cc:729] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae21df6deb645bebd558b6a1a638b40, State: Initialized, Role: FOLLOWER
I20260927 15:16:58.342679 15283 consensus_queue.cc:260] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.343588 15283 ts_tablet_manager.cc:1428] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 15:16:58.343896 15003 heartbeater.cc:503] Master 127.9.242.190:34211 was elected leader, sending a full tablet report...
W20260927 15:16:58.362955 15135 tablet.cc:2369] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:16:58.432044 15004 tablet.cc:2369] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:58.494748 15288 raft_consensus.cc:482] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:16:58.494954 15288 raft_consensus.cc:504] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.496143 15288 leader_election.cc:288] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dae21df6deb645bebd558b6a1a638b40 (127.9.242.129:34235), 695c1e47a7e949c58afa0b2e7d353106 (127.9.242.130:38889)
I20260927 15:16:58.497998 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.496571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:45501 (local address 127.9.242.130:38889)
0927 15:16:58.496692 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.496696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.496709 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:16:58.496744 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.496746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.496773 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.496837 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.497585 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.497589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.497592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.497604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.497625 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.497635 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.497744 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.497819 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.497860 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:16:58.498102 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.496474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:45621 (local address 127.9.242.129:34235)
0927 15:16:58.496611 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.496615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.496856 (+ 241us) server_negotiation.cc:407] Connection header received
0927 15:16:58.496960 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.496962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.496995 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.497054 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.497738 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.497741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.497743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.497756 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.497778 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.497788 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.497868 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.497935 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.497969 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:16:58.498123 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.496401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:34235 (local address 127.9.242.131:45621)
0927 15:16:58.496767 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.496778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.496868 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.497076 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.497078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.497092 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:58.497555 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:58.497557 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:58.497567 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:58.497590 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:58.497860 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:58.497863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:58.497866 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:16:58.497938 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:16:58.497968 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":105,"thread_start_us":113,"threads_started":1}
I20260927 15:16:58.497979 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.496398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:38889 (local address 127.9.242.131:45501)
0927 15:16:58.496500 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.496515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.496603 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.496880 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.496883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.496897 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:16:58.497416 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:16:58.497418 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:16:58.497430 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:16:58.497460 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:16:58.497731 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:16:58.497734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:16:58.497737 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 15:16:58.497815 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:16:58.497851 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 15:16:58.498754 14940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19679c05a8e4ffaa099155472ddc512" candidate_uuid: "efec3528e0054636afd07867513b63ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae21df6deb645bebd558b6a1a638b40" is_pre_election: true
I20260927 15:16:58.498992 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19679c05a8e4ffaa099155472ddc512" candidate_uuid: "efec3528e0054636afd07867513b63ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695c1e47a7e949c58afa0b2e7d353106" is_pre_election: true
I20260927 15:16:58.499039 14940 raft_consensus.cc:2454] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efec3528e0054636afd07867513b63ff in term 0.
I20260927 15:16:58.499274 15088 raft_consensus.cc:2454] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efec3528e0054636afd07867513b63ff in term 0.
I20260927 15:16:58.499502 15152 leader_election.cc:302] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dae21df6deb645bebd558b6a1a638b40, efec3528e0054636afd07867513b63ff; no voters:
I20260927 15:16:58.499782 15288 raft_consensus.cc:2781] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:16:58.499861 15288 raft_consensus.cc:482] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:16:58.499907 15288 raft_consensus.cc:3037] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:58.500890 15288 raft_consensus.cc:504] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.501230 15288 leader_election.cc:288] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [CANDIDATE]: Term 1 election: Requested vote from peers dae21df6deb645bebd558b6a1a638b40 (127.9.242.129:34235), 695c1e47a7e949c58afa0b2e7d353106 (127.9.242.130:38889)
I20260927 15:16:58.501542 14940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19679c05a8e4ffaa099155472ddc512" candidate_uuid: "efec3528e0054636afd07867513b63ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae21df6deb645bebd558b6a1a638b40"
I20260927 15:16:58.501551 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19679c05a8e4ffaa099155472ddc512" candidate_uuid: "efec3528e0054636afd07867513b63ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695c1e47a7e949c58afa0b2e7d353106"
I20260927 15:16:58.501725 15088 raft_consensus.cc:3037] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:58.501727 14940 raft_consensus.cc:3037] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:58.502705 15088 raft_consensus.cc:2454] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efec3528e0054636afd07867513b63ff in term 1.
I20260927 15:16:58.502731 14940 raft_consensus.cc:2454] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efec3528e0054636afd07867513b63ff in term 1.
I20260927 15:16:58.503014 15152 leader_election.cc:302] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 695c1e47a7e949c58afa0b2e7d353106, dae21df6deb645bebd558b6a1a638b40, efec3528e0054636afd07867513b63ff; no voters:
I20260927 15:16:58.503219 15288 raft_consensus.cc:2781] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:58.503386 15288 raft_consensus.cc:686] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 1 LEADER]: Becoming Leader. State: Replica: efec3528e0054636afd07867513b63ff, State: Running, Role: LEADER
I20260927 15:16:58.503698 15288 consensus_queue.cc:237] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } }
I20260927 15:16:58.506770 14815 catalog_manager.cc:5495] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff reported cstate change: term changed from 0 to 1, leader changed from <none> to efec3528e0054636afd07867513b63ff (127.9.242.131). New cstate: current_term: 1 leader_uuid: "efec3528e0054636afd07867513b63ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efec3528e0054636afd07867513b63ff" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 } health_report { overall_health: UNKNOWN } } }
W20260927 15:16:58.528682 15266 tablet.cc:2369] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:16:58.553066 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.551810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.9.242.131:45885)
0927 15:16:58.551926 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.551930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.551942 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:16:58.552039 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.552042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.552099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.552163 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.552724 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.552726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.552729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.552738 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.552756 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.552765 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.552836 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.552891 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.552923 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:16:58.558712 14940 raft_consensus.cc:1261] T b19679c05a8e4ffaa099155472ddc512 P dae21df6deb645bebd558b6a1a638b40 [term 1 FOLLOWER]: Refusing update from remote peer efec3528e0054636afd07867513b63ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:58.558712 15088 raft_consensus.cc:1261] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 1 FOLLOWER]: Refusing update from remote peer efec3528e0054636afd07867513b63ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:16:58.559406 15288 consensus_queue.cc:1035] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "695c1e47a7e949c58afa0b2e7d353106" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:58.559580 15292 consensus_queue.cc:1035] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "dae21df6deb645bebd558b6a1a638b40" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:16:58.567090 15297 mvcc.cc:205] Tried to move back new op lower bound from 7333979007209553920 to 7333979006993637376. Current Snapshot: MvccSnapshot[applied={T|T < 7333979007209553920}]
I20260927 15:16:58.567791 15296 mvcc.cc:205] Tried to move back new op lower bound from 7333979007209553920 to 7333979006993637376. Current Snapshot: MvccSnapshot[applied={T|T < 7333979007209553920}]
I20260927 15:16:58.570029 15295 mvcc.cc:205] Tried to move back new op lower bound from 7333979007209553920 to 7333979006993637376. Current Snapshot: MvccSnapshot[applied={T|T < 7333979007209553920 or (T in {7333979007209553920})}]
I20260927 15:16:58.574668 14814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:58.574808 14814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:16:58.577070 14814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4eb225211245403ebece2a6b2f934e13: Sending DeleteTablet for 3 replicas of tablet b19679c05a8e4ffaa099155472ddc512
I20260927 15:16:58.577787 15199 tablet_service.cc:1514] Processing DeleteTablet for tablet b19679c05a8e4ffaa099155472ddc512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:58 UTC) from {username='slave'} at 127.0.0.1:51520
I20260927 15:16:58.577795 15068 tablet_service.cc:1514] Processing DeleteTablet for tablet b19679c05a8e4ffaa099155472ddc512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:58 UTC) from {username='slave'} at 127.0.0.1:56680
I20260927 15:16:58.577811 14937 tablet_service.cc:1514] Processing DeleteTablet for tablet b19679c05a8e4ffaa099155472ddc512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:16:58 UTC) from {username='slave'} at 127.0.0.1:51774
I20260927 15:16:58.578428 15306 tablet_replica.cc:330] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: stopping tablet replica
I20260927 15:16:58.578724 15306 raft_consensus.cc:2229] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:16:58.578972 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 14875
I20260927 15:16:58.579044 15306 raft_consensus.cc:2258] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:58.579597 15305 tablet_replica.cc:330] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: stopping tablet replica
I20260927 15:16:58.579857 15305 raft_consensus.cc:2229] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:16:58.580199 15305 raft_consensus.cc:2258] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:16:58.581748 15305 ts_tablet_manager.cc:1905] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:58.581748 15306 ts_tablet_manager.cc:1905] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:16:58.584417 15305 ts_tablet_manager.cc:1918] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:58.584417 15306 ts_tablet_manager.cc:1918] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:16:58.584514 15306 log.cc:1196] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/b19679c05a8e4ffaa099155472ddc512
I20260927 15:16:58.584647 15305 log.cc:1196] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/b19679c05a8e4ffaa099155472ddc512
I20260927 15:16:58.584811 15306 ts_tablet_manager.cc:1939] T b19679c05a8e4ffaa099155472ddc512 P 695c1e47a7e949c58afa0b2e7d353106: Deleting consensus metadata
I20260927 15:16:58.585062 15305 ts_tablet_manager.cc:1939] T b19679c05a8e4ffaa099155472ddc512 P efec3528e0054636afd07867513b63ff: Deleting consensus metadata
I20260927 15:16:58.586156 14802 catalog_manager.cc:4841] TS 695c1e47a7e949c58afa0b2e7d353106 (127.9.242.130:38889): tablet b19679c05a8e4ffaa099155472ddc512 (table test-table [id=b6ea107770c944e99dca561771b171f6]) successfully deleted
I20260927 15:16:58.586156 14800 catalog_manager.cc:4841] TS efec3528e0054636afd07867513b63ff (127.9.242.131:45885): tablet b19679c05a8e4ffaa099155472ddc512 (table test-table [id=b6ea107770c944e99dca561771b171f6]) successfully deleted
W20260927 15:16:58.587932 14799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:34235 (error 108)
I20260927 15:16:58.588740 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15006
I20260927 15:16:58.588897 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.588634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:34235 (local address 127.0.0.1:51778)
0927 15:16:58.588750 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.588815 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:34235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 15:16:58.589960 14799 catalog_manager.cc:4584] TS dae21df6deb645bebd558b6a1a638b40 (127.9.242.129:34235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b19679c05a8e4ffaa099155472ddc512: Network error: Client connection negotiation failed: client connection to 127.9.242.129:34235: connect: Connection refused (error 111)
I20260927 15:16:58.593479 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15137
I20260927 15:16:58.598198 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 14784
2026-09-27T15:16:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1073 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 15:16:58.617811 10186 mini_oidc.cc:152] Starting JWKS server
I20260927 15:16:58.619339 10186 webserver.cc:460] Webserver started at https://127.0.0.1:36905/ using document root <none> and password file <none>
I20260927 15:16:58.619498 10186 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:16:58.619740 10186 webserver.cc:460] Webserver started at http://127.0.0.1:42819/ using document root <none> and password file <none>
2026-09-27T15:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:58Z Disabled control of system clock
I20260927 15:16:58.627851 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37409
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:41325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:37409
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/testchainca.cert with env {}
W20260927 15:16:58.734535 15313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:58.734815 15313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:58.734856 15313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:58.738987 15313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:58.739045 15313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:58.739061 15313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:58.739084 15313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:58.739099 15313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:16:58.739116 15313 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:16:58.739142 15313 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36905/jwks.json
I20260927 15:16:58.745000 15313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:41325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:37409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:58.746098 15313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:58.747067 15313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:58.755173 15318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:58.755205 15319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:58.755383 15313 server_base.cc:1034] running on GCE node
W20260927 15:16:58.755203 15321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:58.755986 15313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:58.757059 15313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:58.758224 15313 hybrid_clock.cc:648] HybridClock initialized: now 1790522218758198 us; error 37 us; skew 500 ppm
I20260927 15:16:58.759996 15313 webserver.cc:460] Webserver started at http://127.9.242.190:33327/ using document root <none> and password file <none>
I20260927 15:16:58.760556 15313 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:58.760623 15313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:58.760815 15313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:58.762868 15313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "c32fc9b48025417dbc1f1b34d8571aad"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "aa1c3f2d6b2bbf8ddb4e8d736968a782"
server_key_iv: "5c8af8378245277f30dd463d4f591b72"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.763382 15313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "c32fc9b48025417dbc1f1b34d8571aad"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "aa1c3f2d6b2bbf8ddb4e8d736968a782"
server_key_iv: "5c8af8378245277f30dd463d4f591b72"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.766886 15313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:58.769249 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:58.770535 15313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:58.770675 15313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "c32fc9b48025417dbc1f1b34d8571aad"
format_stamp: "Formatted at 2026-09-27 15:16:58 on dist-test-slave-m4lt"
server_key: "aa1c3f2d6b2bbf8ddb4e8d736968a782"
server_key_iv: "5c8af8378245277f30dd463d4f591b72"
server_key_version: "encryptionkey@0"
I20260927 15:16:58.770771 15313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:58.792239 15313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:58.792912 15313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:58.793073 15313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:58.799400 15313 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:37409
I20260927 15:16:58.799412 15379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:37409 every 8 connection(s)
I20260927 15:16:58.800303 15313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:58.802927 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15313
I20260927 15:16:58.803052 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:58.803340 10186 external_mini_cluster.cc:1428] Setting key 80361507410195a7f164a75943428da8
I20260927 15:16:58.803676 15380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:58.809880 15380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad: Bootstrap starting.
I20260927 15:16:58.812028 15380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:58.812307 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.804740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.9.242.190:37409)
0927 15:16:58.805363 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.805377 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.805414 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:16:58.806239 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.806257 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.806669 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.807012 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:58.807413 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:58.808349 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:58.809200 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:58.809750 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:58.809891 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:58.810667 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:58.810672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:58.810684 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:58.811047 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:58.811103 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:58.811123 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:58.811203 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:58.811331 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 15:16:58.811575 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260927 15:16:58.812831 15380 log.cc:824] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:58.814963 15380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad: No bootstrap required, opened a new log
I20260927 15:16:58.815191 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:16:58.818434 15380 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fc9b48025417dbc1f1b34d8571aad" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37409 } }
I20260927 15:16:58.818645 15380 raft_consensus.cc:374] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:58.818694 15380 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32fc9b48025417dbc1f1b34d8571aad, State: Initialized, Role: FOLLOWER
I20260927 15:16:58.819116 15380 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fc9b48025417dbc1f1b34d8571aad" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37409 } }
I20260927 15:16:58.819262 15380 raft_consensus.cc:388] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:58.819322 15380 raft_consensus.cc:482] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:58.819417 15380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:58.820252 15380 raft_consensus.cc:504] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fc9b48025417dbc1f1b34d8571aad" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37409 } }
I20260927 15:16:58.820533 15380 leader_election.cc:302] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c32fc9b48025417dbc1f1b34d8571aad; no voters:
I20260927 15:16:58.820801 15380 leader_election.cc:288] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:58.820900 15385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:58.821100 15385 raft_consensus.cc:686] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [term 1 LEADER]: Becoming Leader. State: Replica: c32fc9b48025417dbc1f1b34d8571aad, State: Running, Role: LEADER
I20260927 15:16:58.821381 15385 consensus_queue.cc:237] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fc9b48025417dbc1f1b34d8571aad" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37409 } }
I20260927 15:16:58.821943 15380 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:58.823088 15387 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c32fc9b48025417dbc1f1b34d8571aad. Latest consensus state: current_term: 1 leader_uuid: "c32fc9b48025417dbc1f1b34d8571aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fc9b48025417dbc1f1b34d8571aad" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37409 } } }
I20260927 15:16:58.823093 15386 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c32fc9b48025417dbc1f1b34d8571aad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c32fc9b48025417dbc1f1b34d8571aad" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37409 } } }
I20260927 15:16:58.823230 15386 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:58.823227 15387 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:58.823599 15394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:58.826867 15394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:58.832214 15394 catalog_manager.cc:1347] Generated new cluster ID: 3985dcdbca714b12be82607ed91bfd0c
I20260927 15:16:58.832284 15394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:58.839134 15394 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:58.839969 15394 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:58.846695 15394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c32fc9b48025417dbc1f1b34d8571aad: Generated new TSK 0
I20260927 15:16:58.847383 15394 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:16:58.879869 15410 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 15:16:58.879974 15410 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:16:58.880597 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:58.863311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.9.242.190:37409)
0927 15:16:58.863432 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:58.863436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:58.863564 (+ 128us) server_negotiation.cc:407] Connection header received
0927 15:16:58.863754 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:58.863758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:58.863818 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:58.863933 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:16:58.864141 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:58.864970 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:58.865926 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:58.866254 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:58.866354 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:58.866640 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:16:58.880155 (+ 13515us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:16:58.880438 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 15:16:58.880666 15383 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 15:16:58.881871 15409 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:16:58.863185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37409 (local address 127.0.0.1:51410)
0927 15:16:58.863488 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:58.863500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:58.863608 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:58.863933 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:58.863936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:58.863942 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:16:58.864053 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:58.864058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:58.865102 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:58.865106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:58.865826 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:58.865836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:58.866355 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:58.866357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:58.866423 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:58.866467 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:16:58.880423 (+ 13956us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 15:16:58.880671 (+ 248us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.242.190:37409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.242.190:37409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 15:16:58.882653 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15313
2026-09-27T15:16:58Z chronyd exiting
I20260927 15:16:58.918664 10186 test_util.cc:175] -----------------------------------------------
I20260927 15:16:58.918769 10186 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790522194092531-10186-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 15:16:58.920025 10186 mini_oidc.cc:152] Starting JWKS server
I20260927 15:16:58.920861 10186 webserver.cc:460] Webserver started at https://127.0.0.1:37341/ using document root <none> and password file <none>
I20260927 15:16:58.921363 10186 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:16:58.921706 10186 webserver.cc:460] Webserver started at http://127.0.0.1:35277/ using document root <none> and password file <none>
2026-09-27T15:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:58Z Disabled control of system clock
I20260927 15:16:58.930045 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:33913
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:43561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:33913
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37341/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/testchainca.cert with env {}
W20260927 15:16:59.038532 15416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:59.038796 15416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:59.038836 15416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:59.042830 15416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:59.042887 15416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:59.042902 15416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:59.042917 15416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:59.042932 15416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:16:59.042956 15416 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:16:59.042974 15416 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37341/jwks.json
I20260927 15:16:59.048347 15416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:33913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:33913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37341/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:59.049337 15416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:59.050325 15416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:59.058249 15421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:59.058244 15424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:59.058521 15416 server_base.cc:1034] running on GCE node
W20260927 15:16:59.058243 15422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:59.059132 15416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:59.060135 15416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:59.061291 15416 hybrid_clock.cc:648] HybridClock initialized: now 1790522219061269 us; error 39 us; skew 500 ppm
I20260927 15:16:59.063030 15416 webserver.cc:460] Webserver started at http://127.9.242.190:38599/ using document root <none> and password file <none>
I20260927 15:16:59.063565 15416 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:59.063635 15416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:59.063812 15416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:59.065811 15416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "7ec5c679bf60487d8dd362b7255bb845"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "a822477a6c5960c4c0cae576179bc173"
server_key_iv: "e47664ea3b461f96cc26246fb3a2a954"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.066325 15416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "7ec5c679bf60487d8dd362b7255bb845"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "a822477a6c5960c4c0cae576179bc173"
server_key_iv: "e47664ea3b461f96cc26246fb3a2a954"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.069716 15416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:16:59.071923 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:59.073094 15416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:16:59.073197 15416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "7ec5c679bf60487d8dd362b7255bb845"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "a822477a6c5960c4c0cae576179bc173"
server_key_iv: "e47664ea3b461f96cc26246fb3a2a954"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.073284 15416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:59.083014 15416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:59.083633 15416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:59.083792 15416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:59.090353 15416 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:33913
I20260927 15:16:59.090370 15482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:33913 every 8 connection(s)
I20260927 15:16:59.091284 15416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:59.094439 15483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:59.095036 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15416
I20260927 15:16:59.095147 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:59.095399 10186 external_mini_cluster.cc:1428] Setting key 82086d5046734aeeeae0cf5c3db1eb59
I20260927 15:16:59.101217 15483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845: Bootstrap starting.
I20260927 15:16:59.103329 15483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:59.104043 15483 log.cc:824] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:59.104302 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:59.096548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.9.242.190:33913)
0927 15:16:59.097161 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:59.097181 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:59.097217 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:16:59.098128 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:59.098148 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:59.098571 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:59.098886 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:59.099293 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.100289 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.101157 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.101643 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.101821 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.102610 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:59.102616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:59.102629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:59.102994 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:59.103047 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:59.103065 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:59.103149 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:59.103277 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 15:16:59.103546 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260927 15:16:59.105888 15483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845: No bootstrap required, opened a new log
I20260927 15:16:59.106920 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:16:59.109512 15483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec5c679bf60487d8dd362b7255bb845" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33913 } }
I20260927 15:16:59.109761 15483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:59.109803 15483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec5c679bf60487d8dd362b7255bb845, State: Initialized, Role: FOLLOWER
I20260927 15:16:59.110183 15483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec5c679bf60487d8dd362b7255bb845" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33913 } }
I20260927 15:16:59.110294 15483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:59.110337 15483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:59.110402 15483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:59.111191 15483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec5c679bf60487d8dd362b7255bb845" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33913 } }
I20260927 15:16:59.111467 15483 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec5c679bf60487d8dd362b7255bb845; no voters:
I20260927 15:16:59.111713 15483 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:59.111811 15488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:59.111999 15488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec5c679bf60487d8dd362b7255bb845, State: Running, Role: LEADER
I20260927 15:16:59.112327 15488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec5c679bf60487d8dd362b7255bb845" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33913 } }
I20260927 15:16:59.112762 15483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:59.113803 15490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec5c679bf60487d8dd362b7255bb845. Latest consensus state: current_term: 1 leader_uuid: "7ec5c679bf60487d8dd362b7255bb845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec5c679bf60487d8dd362b7255bb845" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33913 } } }
I20260927 15:16:59.113924 15490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:59.114223 15489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec5c679bf60487d8dd362b7255bb845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec5c679bf60487d8dd362b7255bb845" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33913 } } }
I20260927 15:16:59.114329 15489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:59.114589 15497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:59.117101 15497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:59.121778 15497 catalog_manager.cc:1347] Generated new cluster ID: 356e858059a04186be4923095b08cbf3
I20260927 15:16:59.121838 15497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:59.145155 15497 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:59.146202 15497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:59.154910 15497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec5c679bf60487d8dd362b7255bb845: Generated new TSK 0
I20260927 15:16:59.155670 15497 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 15:16:59.180877 15513 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 15:16:59.180963 15513 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 15:16:59.181558 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:59.166055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.9.242.190:33913)
0927 15:16:59.166177 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:59.166182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:59.166360 (+ 178us) server_negotiation.cc:407] Connection header received
0927 15:16:59.166474 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:59.166476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:59.166525 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:59.166618 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:16:59.166843 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.167428 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.168637 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.168858 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.168934 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.169248 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:16:59.181223 (+ 11975us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:16:59.181438 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:16:59.181612 15486 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 15:16:59.181788 15512 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:16:59.165935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:33913 (local address 127.0.0.1:36014)
0927 15:16:59.166237 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:59.166251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:59.166363 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:59.166627 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:59.166630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:59.166639 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:16:59.166749 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:59.166754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.167521 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.167524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:59.168516 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:59.168526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.168967 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.168971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:59.169060 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.169088 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:16:59.181400 (+ 12312us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 15:16:59.181583 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.242.190:33913: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":109,"threads_started":1}
I20260927 15:16:59.182365 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15416
2026-09-27T15:16:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 15:16:59.220589 10186 mini_oidc.cc:152] Starting JWKS server
I20260927 15:16:59.221421 10186 webserver.cc:460] Webserver started at https://127.0.0.1:36971/ using document root <none> and password file <none>
I20260927 15:16:59.221509 10186 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 15:16:59.221783 10186 webserver.cc:460] Webserver started at http://127.0.0.1:42921/ using document root <none> and password file <none>
2026-09-27T15:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:59Z Disabled control of system clock
I20260927 15:16:59.230191 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:33861
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:33861
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36971/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:16:59.333899 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:59.334167 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:59.334204 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:59.338104 15519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:59.338157 15519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:59.338171 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:59.338186 15519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:59.338199 15519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:16:59.338213 15519 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:16:59.338236 15519 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36971/jwks.json
I20260927 15:16:59.342897 15519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:33861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36971/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:59.343894 15519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:59.345235 15519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:59.353222 15527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:59.353231 15524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:59.353279 15519 server_base.cc:1034] running on GCE node
W20260927 15:16:59.353230 15525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:59.354019 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:59.355037 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:59.356202 15519 hybrid_clock.cc:648] HybridClock initialized: now 1790522219356178 us; error 40 us; skew 500 ppm
I20260927 15:16:59.357956 15519 webserver.cc:460] Webserver started at http://127.9.242.190:33017/ using document root <none> and password file <none>
I20260927 15:16:59.358485 15519 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:59.358547 15519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:59.358752 15519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:59.360850 15519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "6288c40d1a62424a90871ed2558bfda9"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "54822534a1c058727e850a4279482e9d"
server_key_iv: "4e88b36a7c2249112235dd910093a85e"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.361389 15519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "6288c40d1a62424a90871ed2558bfda9"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "54822534a1c058727e850a4279482e9d"
server_key_iv: "4e88b36a7c2249112235dd910093a85e"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.365016 15519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:16:59.367513 15533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:59.368860 15519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:16:59.368973 15519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "6288c40d1a62424a90871ed2558bfda9"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "54822534a1c058727e850a4279482e9d"
server_key_iv: "4e88b36a7c2249112235dd910093a85e"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.369076 15519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:59.384821 15519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:59.385493 15519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:59.385695 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:59.392285 15519 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:33861
I20260927 15:16:59.392311 15585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:33861 every 8 connection(s)
I20260927 15:16:59.393215 15519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:59.395181 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15519
I20260927 15:16:59.395295 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:59.395567 10186 external_mini_cluster.cc:1428] Setting key 7ea80f1e8bea725854af2068536204b7
I20260927 15:16:59.396606 15586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:59.402473 15586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9: Bootstrap starting.
I20260927 15:16:59.404469 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:59.396874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.9.242.190:33861)
0927 15:16:59.397450 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:59.397465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:59.397500 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:16:59.398137 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:59.398153 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:59.398543 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:59.398819 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:59.399169 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.400094 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.401259 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.401739 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.401853 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.402658 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:59.402663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:59.402675 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:59.403046 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:59.403098 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:59.403127 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:59.403196 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:59.403383 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 15:16:59.403670 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260927 15:16:59.404771 15586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:59.405433 15586 log.cc:824] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:59.407191 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:16:59.407403 15586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9: No bootstrap required, opened a new log
I20260927 15:16:59.411095 15586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288c40d1a62424a90871ed2558bfda9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33861 } }
I20260927 15:16:59.411314 15586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:59.411376 15586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6288c40d1a62424a90871ed2558bfda9, State: Initialized, Role: FOLLOWER
I20260927 15:16:59.411773 15586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288c40d1a62424a90871ed2558bfda9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33861 } }
I20260927 15:16:59.411893 15586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:59.411952 15586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:59.412051 15586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:59.412889 15586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288c40d1a62424a90871ed2558bfda9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33861 } }
I20260927 15:16:59.413197 15586 leader_election.cc:302] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6288c40d1a62424a90871ed2558bfda9; no voters:
I20260927 15:16:59.413470 15586 leader_election.cc:288] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:59.413548 15591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:59.413756 15591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [term 1 LEADER]: Becoming Leader. State: Replica: 6288c40d1a62424a90871ed2558bfda9, State: Running, Role: LEADER
I20260927 15:16:59.414052 15591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288c40d1a62424a90871ed2558bfda9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33861 } }
I20260927 15:16:59.414649 15586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:59.415796 15593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6288c40d1a62424a90871ed2558bfda9. Latest consensus state: current_term: 1 leader_uuid: "6288c40d1a62424a90871ed2558bfda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288c40d1a62424a90871ed2558bfda9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33861 } } }
I20260927 15:16:59.415807 15592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6288c40d1a62424a90871ed2558bfda9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288c40d1a62424a90871ed2558bfda9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33861 } } }
I20260927 15:16:59.415974 15592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:59.416178 15593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:59.416831 15601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:59.419498 15601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:59.425364 15601 catalog_manager.cc:1347] Generated new cluster ID: b8dcf49c410e49eaa84735851641e794
I20260927 15:16:59.425436 15601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:59.454007 15601 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:59.455257 15601 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:59.468406 15601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6288c40d1a62424a90871ed2558bfda9: Generated new TSK 0
I20260927 15:16:59.469249 15601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:16:59.483331 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:59.478467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.9.242.190:33861)
0927 15:16:59.478575 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:59.478580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:59.478701 (+ 121us) server_negotiation.cc:407] Connection header received
0927 15:16:59.478825 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:59.478828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:59.478883 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:59.478993 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:16:59.479168 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.479958 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.480794 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.481134 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.481229 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.481496 (+ 267us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:16:59.482959 (+ 1463us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:16:59.483187 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 15:16:59.483400 15589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 15:16:59.484913 15615 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:16:59.478362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:33861 (local address 127.0.0.1:57376)
0927 15:16:59.478618 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:59.478631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:59.478724 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:59.478987 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:59.478989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:59.478995 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:16:59.479077 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:59.479081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.480048 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.480050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:59.480694 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:59.480702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.481247 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.481250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:59.481315 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.481338 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:16:59.483119 (+ 1781us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 15:16:59.484748 (+ 1629us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.242.190:33861: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36971/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1461,"thread_start_us":87,"threads_started":1}
I20260927 15:16:59.485589 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15519
E20260927 15:16:59.493916 15616 webserver.cc:533] Webserver: sslize failed:
E20260927 15:16:59.493981 15616 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T15:16:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T15:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:16:59Z Disabled control of system clock
I20260927 15:16:59.530788 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34315
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:46593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:34315
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/testchainca.cert with env {}
W20260927 15:16:59.635753 15620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:16:59.635996 15620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:16:59.636041 15620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:16:59.639905 15620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:16:59.639959 15620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:16:59.639972 15620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:16:59.639988 15620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:16:59.640002 15620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 15:16:59.640017 15620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 15:16:59.640040 15620 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 15:16:59.645231 15620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:46593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:34315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:16:59.646211 15620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:16:59.647123 15620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:16:59.655074 15625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:16:59.655061 15628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:59.655150 15620 server_base.cc:1034] running on GCE node
W20260927 15:16:59.655061 15626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:16:59.655826 15620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:16:59.656893 15620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:16:59.658133 15620 hybrid_clock.cc:648] HybridClock initialized: now 1790522219658099 us; error 45 us; skew 500 ppm
I20260927 15:16:59.659858 15620 webserver.cc:460] Webserver started at http://127.9.242.190:42987/ using document root <none> and password file <none>
I20260927 15:16:59.660485 15620 fs_manager.cc:359] Metadata directory not provided
I20260927 15:16:59.660555 15620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:16:59.660738 15620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:16:59.662761 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "36765ebddaf54928876d5e4ecf4f9504"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "d484eaa7c9e5a40a35e0bcce272913fc"
server_key_iv: "29f8f70f75321ffc91fb59f6e19375c9"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.663316 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "36765ebddaf54928876d5e4ecf4f9504"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "d484eaa7c9e5a40a35e0bcce272913fc"
server_key_iv: "29f8f70f75321ffc91fb59f6e19375c9"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.666851 15620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:16:59.669348 15634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:16:59.670604 15620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:16:59.670722 15620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "36765ebddaf54928876d5e4ecf4f9504"
format_stamp: "Formatted at 2026-09-27 15:16:59 on dist-test-slave-m4lt"
server_key: "d484eaa7c9e5a40a35e0bcce272913fc"
server_key_iv: "29f8f70f75321ffc91fb59f6e19375c9"
server_key_version: "encryptionkey@0"
I20260927 15:16:59.670814 15620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:16:59.694710 15620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:16:59.695451 15620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:16:59.695618 15620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:16:59.701951 15620 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:34315
I20260927 15:16:59.701972 15686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:34315 every 8 connection(s)
I20260927 15:16:59.702857 15620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:16:59.705564 15687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:16:59.705763 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15620
I20260927 15:16:59.705865 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:16:59.706127 10186 external_mini_cluster.cc:1428] Setting key feaec08de3cf8e201fca96e40d0339d6
I20260927 15:16:59.711894 15687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504: Bootstrap starting.
I20260927 15:16:59.714330 15687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504: Neither blocks nor log segments found. Creating new log.
I20260927 15:16:59.714965 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:59.707413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.9.242.190:34315)
0927 15:16:59.707983 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:59.707997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:59.708032 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:16:59.708691 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:59.708708 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:59.709085 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:59.709357 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:16:59.709710 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.710640 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.711843 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.712384 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.712503 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.713321 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:16:59.713325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:16:59.713338 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:16:59.713741 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:16:59.713795 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:16:59.713815 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:16:59.713886 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:16:59.714018 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 15:16:59.714266 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260927 15:16:59.715003 15687 log.cc:824] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504: Log is configured to *not* fsync() on all Append() calls
I20260927 15:16:59.716811 15687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504: No bootstrap required, opened a new log
I20260927 15:16:59.717634 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:16:59.720496 15687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36765ebddaf54928876d5e4ecf4f9504" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34315 } }
I20260927 15:16:59.720721 15687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:16:59.720773 15687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36765ebddaf54928876d5e4ecf4f9504, State: Initialized, Role: FOLLOWER
I20260927 15:16:59.721168 15687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36765ebddaf54928876d5e4ecf4f9504" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34315 } }
I20260927 15:16:59.721288 15687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:16:59.721349 15687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:16:59.721441 15687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:16:59.722332 15687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36765ebddaf54928876d5e4ecf4f9504" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34315 } }
I20260927 15:16:59.722653 15687 leader_election.cc:302] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36765ebddaf54928876d5e4ecf4f9504; no voters:
I20260927 15:16:59.722946 15687 leader_election.cc:288] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:16:59.723063 15692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:16:59.723278 15692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [term 1 LEADER]: Becoming Leader. State: Replica: 36765ebddaf54928876d5e4ecf4f9504, State: Running, Role: LEADER
I20260927 15:16:59.723555 15692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36765ebddaf54928876d5e4ecf4f9504" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34315 } }
I20260927 15:16:59.724094 15687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:16:59.724974 15694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36765ebddaf54928876d5e4ecf4f9504. Latest consensus state: current_term: 1 leader_uuid: "36765ebddaf54928876d5e4ecf4f9504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36765ebddaf54928876d5e4ecf4f9504" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34315 } } }
I20260927 15:16:59.725103 15694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:59.725400 15693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36765ebddaf54928876d5e4ecf4f9504" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36765ebddaf54928876d5e4ecf4f9504" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34315 } } }
I20260927 15:16:59.725515 15693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504 [sys.catalog]: This master's current role is: LEADER
I20260927 15:16:59.725879 15701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:16:59.728685 15701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:16:59.733641 15701 catalog_manager.cc:1347] Generated new cluster ID: 2ec8e58966ef462dac06136eeaef57a9
I20260927 15:16:59.733731 15701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:16:59.742861 15701 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:16:59.743731 15701 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:16:59.750283 15701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36765ebddaf54928876d5e4ecf4f9504: Generated new TSK 0
I20260927 15:16:59.750911 15701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:17:00.185962 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:16:59.760990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.9.242.190:34315)
0927 15:16:59.761110 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:16:59.761115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:16:59.761263 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:16:59.761409 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:16:59.761411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:16:59.761455 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:16:59.761539 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 15:16:59.761779 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.762295 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.763484 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.763745 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.763819 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.764126 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 15:17:00.185533 (+421407us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:17:00.185841 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:17:00.186034 15690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 15:17:00.186411 15716 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:16:59.760919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34315 (local address 127.0.0.1:52870)
0927 15:16:59.761182 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:16:59.761196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:16:59.761309 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:16:59.761562 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:16:59.761565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:16:59.761573 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 15:16:59.761697 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:59.761703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.762393 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.762397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:59.763364 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:16:59.763374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:16:59.763851 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:16:59.763855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:16:59.763946 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:16:59.763974 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 15:17:00.185773 (+421799us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 15:17:00.185954 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.242.190:34315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":87,"threads_started":1}
I20260927 15:17:00.186916 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15620
2026-09-27T15:17:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (681 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (118 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:00 dist-test-slave-m4lt krb5kdc[15728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:00 dist-test-slave-m4lt krb5kdc[15728](info): set up 2 sockets
Sep 27 15:17:00 dist-test-slave-m4lt krb5kdc[15728](info): commencing operation
krb5kdc: starting...
W20260927 15:17:01.389811 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:17:01 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:17:01.504362 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43985
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:43985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:01.616084 15741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:01.616606 15741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:01.616724 15741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:01.621481 15741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:01.621559 15741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:01.621579 15741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:01.621596 15741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:01.621613 15741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:01.627197 15741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:43985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:01.628249 15741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:01.629251 15741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:01.637766 15749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:01.637807 15741 server_base.cc:1034] running on GCE node
W20260927 15:17:01.637766 15747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:01.637801 15746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:01.638622 15741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:01.639779 15741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:01.640950 15741 hybrid_clock.cc:648] HybridClock initialized: now 1790522221640938 us; error 27 us; skew 500 ppm
Sep 27 15:17:01 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522221, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:01.649945 15741 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:17:01.650944 15741 webserver.cc:460] Webserver started at http://127.9.242.190:35551/ using document root <none> and password file <none>
I20260927 15:17:01.651487 15741 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:01.651548 15741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:01.651721 15741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:01.653880 15741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "767355c111ac4063abc8bde198c55221"
format_stamp: "Formatted at 2026-09-27 15:17:01 on dist-test-slave-m4lt"
server_key: "ab0e2ab0daa22a60ac2951862d9c0264"
server_key_iv: "73607301b698b70eb50214400394cc96"
server_key_version: "encryptionkey@0"
I20260927 15:17:01.654400 15741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "767355c111ac4063abc8bde198c55221"
format_stamp: "Formatted at 2026-09-27 15:17:01 on dist-test-slave-m4lt"
server_key: "ab0e2ab0daa22a60ac2951862d9c0264"
server_key_iv: "73607301b698b70eb50214400394cc96"
server_key_version: "encryptionkey@0"
I20260927 15:17:01.657994 15741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:01.660446 15756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:01.661847 15741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:01.661962 15741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "767355c111ac4063abc8bde198c55221"
format_stamp: "Formatted at 2026-09-27 15:17:01 on dist-test-slave-m4lt"
server_key: "ab0e2ab0daa22a60ac2951862d9c0264"
server_key_iv: "73607301b698b70eb50214400394cc96"
server_key_version: "encryptionkey@0"
I20260927 15:17:01.662137 15741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:01.689797 15741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:01.692713 15741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:01.692883 15741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:01.699472 15741 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:43985
I20260927 15:17:01.699564 15808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:43985 every 8 connection(s)
I20260927 15:17:01.700387 15741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:01.703069 15809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:01.708312 15809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221: Bootstrap starting.
I20260927 15:17:01.710557 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15741
I20260927 15:17:01.710673 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:17:01.710781 15809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:01.710945 10186 external_mini_cluster.cc:1428] Setting key 8124009af088004a86037bac07b6284e
I20260927 15:17:01.711612 15809 log.cc:824] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:01.713703 15809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221: No bootstrap required, opened a new log
I20260927 15:17:01.717166 15809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767355c111ac4063abc8bde198c55221" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43985 } }
I20260927 15:17:01.717424 15809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:01.717485 15809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767355c111ac4063abc8bde198c55221, State: Initialized, Role: FOLLOWER
I20260927 15:17:01.718139 15809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767355c111ac4063abc8bde198c55221" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43985 } }
I20260927 15:17:01.718295 15809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:01.718362 15809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:01.718463 15809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 15:17:01 dist-test-slave-m4lt krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:01.719743 15809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767355c111ac4063abc8bde198c55221" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43985 } }
I20260927 15:17:01.720134 15809 leader_election.cc:302] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 767355c111ac4063abc8bde198c55221; no voters:
I20260927 15:17:01.720435 15809 leader_election.cc:288] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:01.720532 15814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:01.720775 15814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [term 1 LEADER]: Becoming Leader. State: Replica: 767355c111ac4063abc8bde198c55221, State: Running, Role: LEADER
I20260927 15:17:01.721139 15814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767355c111ac4063abc8bde198c55221" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43985 } }
I20260927 15:17:01.721720 15809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:01.722854 15816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 767355c111ac4063abc8bde198c55221. Latest consensus state: current_term: 1 leader_uuid: "767355c111ac4063abc8bde198c55221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767355c111ac4063abc8bde198c55221" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43985 } } }
I20260927 15:17:01.722968 15816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:01.723358 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "767355c111ac4063abc8bde198c55221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767355c111ac4063abc8bde198c55221" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43985 } } }
I20260927 15:17:01.723496 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:01.725912 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:01.712036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.9.242.190:43985)
0927 15:17:01.712378 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:01.712393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:01.712429 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:17:01.713108 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:01.713126 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:01.713440 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:01.713799 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:01.714580 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:01.715595 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:01.716410 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:01.716876 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:01.717019 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:01.720082 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:01.720088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:01.720103 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:01.720133 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:01.722151 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:01.722695 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:01.722701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:01.722708 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:01.722857 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:01.723162 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:01.723166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:01.723168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:01.723711 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:01.723905 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:01.724644 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 15:17:01.724932 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
W20260927 15:17:01.726851 15829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:17:01.726925 15829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:17:01.726987 15823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:01.727814 15823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:01.733776 15823 catalog_manager.cc:1347] Generated new cluster ID: e51f77e4334a4bbcb7eb53dc1de85e46
I20260927 15:17:01.733851 15823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:01.763036 15823 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:01.764344 15823 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:01.776170 15823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 767355c111ac4063abc8bde198c55221: Generated new TSK 0
I20260927 15:17:01.776876 15823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:17:01.805018 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43985
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:01.917784 15835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:01.918032 15835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:01.918078 15835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:01.922076 15835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:01.922135 15835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:01.922195 15835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:01.927425 15835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:01.928474 15835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:01.929451 15835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:01.938872 15840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:01.938882 15843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:01.938994 15835 server_base.cc:1034] running on GCE node
W20260927 15:17:01.938889 15841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:01.939559 15835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:01.940222 15835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:01.941391 15835 hybrid_clock.cc:648] HybridClock initialized: now 1790522221941368 us; error 34 us; skew 500 ppm
Sep 27 15:17:01 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522221, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:01.944097 15835 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:17:01.944945 15835 webserver.cc:460] Webserver started at http://127.9.242.129:41165/ using document root <none> and password file <none>
I20260927 15:17:01.945472 15835 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:01.945535 15835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:01.945802 15835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:01.947818 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "c3d0b3621b934348b91a053955707ed6"
format_stamp: "Formatted at 2026-09-27 15:17:01 on dist-test-slave-m4lt"
server_key: "26f3d578b1a29154a0badf216d400e21"
server_key_iv: "8e8b86963faf9bbb4ffae29bd72cce17"
server_key_version: "encryptionkey@0"
I20260927 15:17:01.948348 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "c3d0b3621b934348b91a053955707ed6"
format_stamp: "Formatted at 2026-09-27 15:17:01 on dist-test-slave-m4lt"
server_key: "26f3d578b1a29154a0badf216d400e21"
server_key_iv: "8e8b86963faf9bbb4ffae29bd72cce17"
server_key_version: "encryptionkey@0"
I20260927 15:17:01.952068 15835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 15:17:01.954537 15850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:01.955744 15835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:17:01.955856 15835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "c3d0b3621b934348b91a053955707ed6"
format_stamp: "Formatted at 2026-09-27 15:17:01 on dist-test-slave-m4lt"
server_key: "26f3d578b1a29154a0badf216d400e21"
server_key_iv: "8e8b86963faf9bbb4ffae29bd72cce17"
server_key_version: "encryptionkey@0"
I20260927 15:17:01.955955 15835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:01.972960 15835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:01.976078 15835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:01.976251 15835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:01.976807 15835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:01.977602 15835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:01.977681 15835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:01.977746 15835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:01.977777 15835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:01.985778 15835 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:42683
I20260927 15:17:01.985807 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:42683 every 8 connection(s)
I20260927 15:17:01.986629 15835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:01.991223 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15835
I20260927 15:17:01.991354 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:01.991631 10186 external_mini_cluster.cc:1428] Setting key 0cd9ff529b88bb7e8a90f50b476a240b
Sep 27 15:17:01 dist-test-slave-m4lt krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522221, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:17:02.000098 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:01.988450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:46373 (local address 127.9.242.190:43985)
0927 15:17:01.988622 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:01.988627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:01.989462 (+ 835us) server_negotiation.cc:407] Connection header received
0927 15:17:01.990851 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:01.990854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:01.990903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:01.990992 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:01.992485 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:01.993013 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:01.994190 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:01.994415 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:01.994501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:01.996798 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:01.996802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:01.996805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:01.996830 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:01.998128 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:01.998604 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:01.998608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:01.998610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:01.998685 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:01.998993 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:01.998997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:01.998998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:01.999187 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:01.999270 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:01.999805 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 15:17:01.999932 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:17:02.000949 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:01.988863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43985 (local address 127.9.242.129:46373)
0927 15:17:01.989311 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:01.989352 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:01.990649 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:01.991196 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:01.991207 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:01.991728 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:01.992349 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:01.992368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:01.993133 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:01.993136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:01.994084 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:01.994093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:01.994544 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:01.994548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:01.994724 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:01.995262 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:01.995278 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:01.996642 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:01.998239 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:01.998245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:01.998263 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:01.998514 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:01.998767 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:01.998769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:01.998772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:01.998910 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:01.999284 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:01.999291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:01.999562 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 15:17:01.999746 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 15:17:02.000080 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:17:02.002306 15964 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43985
I20260927 15:17:02.002549 15964 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:02.003058 15964 heartbeater.cc:511] Master 127.9.242.190:43985 requested a full tablet report, sending...
I20260927 15:17:02.004793 15773 ts_manager.cc:194] Registered new tserver with Master: c3d0b3621b934348b91a053955707ed6 (127.9.242.129:42683)
I20260927 15:17:02.006371 15773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:46373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:17:02.016888 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43985
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:02.128830 15969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:02.129138 15969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:02.129184 15969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:02.133396 15969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:02.133462 15969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:02.133527 15969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:02.139010 15969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:02.140093 15969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:02.141094 15969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:02.151114 15975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:02.151050 15977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:02.151063 15974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:02.151239 15969 server_base.cc:1034] running on GCE node
I20260927 15:17:02.151810 15969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:02.152556 15969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:02.153718 15969 hybrid_clock.cc:648] HybridClock initialized: now 1790522222153693 us; error 37 us; skew 500 ppm
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.156380 15969 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:17:02.157193 15969 webserver.cc:460] Webserver started at http://127.9.242.130:42033/ using document root <none> and password file <none>
I20260927 15:17:02.157763 15969 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:02.157825 15969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:02.158010 15969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:02.160045 15969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "f004107f4bf442ef93b578884135e80c"
format_stamp: "Formatted at 2026-09-27 15:17:02 on dist-test-slave-m4lt"
server_key: "ed126ae10b5acf30a4d3cf249a55f272"
server_key_iv: "e15ab350607a5accf6e225be64777ff3"
server_key_version: "encryptionkey@0"
I20260927 15:17:02.160576 15969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "f004107f4bf442ef93b578884135e80c"
format_stamp: "Formatted at 2026-09-27 15:17:02 on dist-test-slave-m4lt"
server_key: "ed126ae10b5acf30a4d3cf249a55f272"
server_key_iv: "e15ab350607a5accf6e225be64777ff3"
server_key_version: "encryptionkey@0"
I20260927 15:17:02.164203 15969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:17:02.166637 15984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:02.167979 15969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:02.168082 15969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "f004107f4bf442ef93b578884135e80c"
format_stamp: "Formatted at 2026-09-27 15:17:02 on dist-test-slave-m4lt"
server_key: "ed126ae10b5acf30a4d3cf249a55f272"
server_key_iv: "e15ab350607a5accf6e225be64777ff3"
server_key_version: "encryptionkey@0"
I20260927 15:17:02.168192 15969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:02.202783 15969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:02.206080 15969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:02.206236 15969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:02.206770 15969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:02.207561 15969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:02.207615 15969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:02.207655 15969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:02.207669 15969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:02.215899 15969 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:44015
I20260927 15:17:02.215945 16097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:44015 every 8 connection(s)
I20260927 15:17:02.216810 15969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:02.223466 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 15969
I20260927 15:17:02.223601 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:02.223894 10186 external_mini_cluster.cc:1428] Setting key c73840cb2170e51a8ef9e50eb07fd858
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:02.231868 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:02.218641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:50095 (local address 127.9.242.190:43985)
0927 15:17:02.218802 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.218806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.219689 (+ 883us) server_negotiation.cc:407] Connection header received
0927 15:17:02.221347 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.221350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.221407 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.221498 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.222754 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.223343 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.224696 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.225006 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.225110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:02.227771 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:02.227775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:02.227778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:02.227807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:02.229310 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.229912 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.229915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.229917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.230003 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.230432 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.230435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.230437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.230684 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:02.230769 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:02.231625 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 15:17:02.231737 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 15:17:02.233039 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:02.219077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43985 (local address 127.9.242.130:50095)
0927 15:17:02.219531 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:02.219575 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:02.221121 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:02.221704 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:02.221711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:02.222098 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:02.222605 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:02.222619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.223473 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.223475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:02.224594 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:02.224603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.225173 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.225178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:02.225393 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:02.225982 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:02.225999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:02.227581 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:02.229441 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:02.229446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:02.229460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:02.229796 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:02.230093 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:02.230097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:02.230099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:02.230300 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:02.230792 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:02.230800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:02.231272 (+ 472us) client_negotiation.cc:769] Sending connection context
0927 15:17:02.231506 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 15:17:02.231899 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:17:02.234421 16098 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43985
I20260927 15:17:02.234656 16098 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:02.235132 16098 heartbeater.cc:511] Master 127.9.242.190:43985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:17:02.236150 15773 ts_manager.cc:194] Registered new tserver with Master: f004107f4bf442ef93b578884135e80c (127.9.242.130:44015)
I20260927 15:17:02.236733 15773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:50095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:17:02.250659 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43985
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:02.357522 16103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:02.357825 16103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:02.357887 16103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:02.361953 16103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:02.362012 16103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:02.362088 16103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:02.367352 16103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:02.368371 16103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:02.369344 16103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:02.378713 16111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:02.378714 16108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:02.378706 16109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:02.378813 16103 server_base.cc:1034] running on GCE node
I20260927 15:17:02.379427 16103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:02.380081 16103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:02.381253 16103 hybrid_clock.cc:648] HybridClock initialized: now 1790522222381230 us; error 36 us; skew 500 ppm
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.383996 16103 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:17:02.384902 16103 webserver.cc:460] Webserver started at http://127.9.242.131:44313/ using document root <none> and password file <none>
I20260927 15:17:02.385429 16103 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:02.385490 16103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:02.385710 16103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:02.387742 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "6de6de09170c4af89f7eaf14ae88ca67"
format_stamp: "Formatted at 2026-09-27 15:17:02 on dist-test-slave-m4lt"
server_key: "33dd24fa85de355fda5ebb21a07a8b50"
server_key_iv: "f48a7373a27344500c32cdc09149166e"
server_key_version: "encryptionkey@0"
I20260927 15:17:02.388285 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "6de6de09170c4af89f7eaf14ae88ca67"
format_stamp: "Formatted at 2026-09-27 15:17:02 on dist-test-slave-m4lt"
server_key: "33dd24fa85de355fda5ebb21a07a8b50"
server_key_iv: "f48a7373a27344500c32cdc09149166e"
server_key_version: "encryptionkey@0"
I20260927 15:17:02.391881 16103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 15:17:02.394435 16118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:02.395721 16103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:02.395826 16103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "6de6de09170c4af89f7eaf14ae88ca67"
format_stamp: "Formatted at 2026-09-27 15:17:02 on dist-test-slave-m4lt"
server_key: "33dd24fa85de355fda5ebb21a07a8b50"
server_key_iv: "f48a7373a27344500c32cdc09149166e"
server_key_version: "encryptionkey@0"
I20260927 15:17:02.395918 16103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:02.413947 16103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:02.417233 16103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:02.417416 16103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:02.418038 16103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:02.418867 16103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:02.418923 16103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:02.418977 16103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:02.419009 16103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:02.427482 16103 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:38313
I20260927 15:17:02.427522 16231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:38313 every 8 connection(s)
I20260927 15:17:02.428349 16103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:02.436575 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 16103
I20260927 15:17:02.436681 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:02.436914 10186 external_mini_cluster.cc:1428] Setting key 19f70ed0aff41f75f074910b8a50a17a
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:02.441424 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:02.430028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:48519 (local address 127.9.242.190:43985)
0927 15:17:02.430224 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.430227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.431049 (+ 822us) server_negotiation.cc:407] Connection header received
0927 15:17:02.432348 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.432351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.432396 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.432466 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.433636 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.434223 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.435433 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.435673 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.435799 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:02.438279 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:02.438282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:02.438284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:02.438307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:02.439605 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.440050 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.440053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.440054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.440127 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.440419 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.440423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.440424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.440603 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:02.440683 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:02.441147 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 15:17:02.441272 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:17:02.442246 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:02.430472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43985 (local address 127.9.242.131:48519)
0927 15:17:02.430896 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:02.430939 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:02.432158 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:02.432612 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:02.432621 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:02.432955 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:02.433480 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:02.433495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.434340 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.434343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:02.435309 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:02.435318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.435841 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.435843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:02.435987 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:02.436588 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:02.436604 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:02.438119 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:02.439706 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:02.439712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:02.439727 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:02.439961 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:02.440204 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:02.440208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:02.440210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:02.440339 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:02.440692 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:02.440699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:02.440932 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 15:17:02.441103 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:17:02.441408 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260927 15:17:02.443288 16232 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43985
I20260927 15:17:02.443519 16232 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:02.444077 16232 heartbeater.cc:511] Master 127.9.242.190:43985 requested a full tablet report, sending...
I20260927 15:17:02.445119 15773 ts_manager.cc:194] Registered new tserver with Master: 6de6de09170c4af89f7eaf14ae88ca67 (127.9.242.131:38313)
I20260927 15:17:02.445876 15773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:48519
I20260927 15:17:02.450932 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/test-admin.keytab.
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.463927 10186 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 15:17:02.466812 16239 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:02.465297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.9.242.129:42683)
0927 15:17:02.465594 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.465608 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.465632 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:17:02.465715 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.465722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.465916 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.466074 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.466629 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53580: BlockingRecv error: recv got EOF from 127.0.0.1:53580 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
W20260927 15:17:02.466833 10186 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.467828 10186 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.468703 10186 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:17:02.478045 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:02.469549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.9.242.129:42683)
0927 15:17:02.469671 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.469676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.469775 (+ 99us) server_negotiation.cc:407] Connection header received
0927 15:17:02.469885 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.469888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.469928 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.469991 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.470508 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.471382 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.472272 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.472638 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.472735 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:02.474960 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:02.474968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:02.474981 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:02.475005 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:02.476402 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.476796 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.476801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.476802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.476876 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.477109 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.477112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.477114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.477356 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:02.477549 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:02.477760 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:17:02.477915 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 15:17:02.481714 16239 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:02.480849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.9.242.129:42683)
0927 15:17:02.480990 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.480993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.481110 (+ 117us) server_negotiation.cc:407] Connection header received
0927 15:17:02.481143 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.481145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.481181 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.481234 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.481578 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53598: BlockingRecv error: recv got EOF from 127.0.0.1:53598 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 15:17:02.481838 10186 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.482793 10186 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.483628 10186 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:17:02.491701 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:02.484432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.9.242.129:42683)
0927 15:17:02.484536 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.484541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.484608 (+ 67us) server_negotiation.cc:407] Connection header received
0927 15:17:02.484762 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.484764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.484802 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.484860 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.485327 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.485931 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.486708 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.486938 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.487024 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:02.489092 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:02.489095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:02.489096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:02.489119 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:02.490309 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.490665 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.490669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.490670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.490744 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.490969 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.490972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.490974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.491127 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:02.491249 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:02.491447 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 15:17:02.491549 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 15:17:02.494266 16239 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:02.493429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.9.242.129:42683)
0927 15:17:02.493574 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.493578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.493620 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:17:02.493735 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.493739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.493781 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.493844 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.494168 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53612: BlockingRecv error: recv got EOF from 127.0.0.1:53612 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 15:17:02.494372 10186 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790522194092531-10186-0/krb5kdc/krb5cc' of type FILE
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.495332 10186 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:02.496176 10186 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:17:02.504410 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:02.496959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.9.242.129:42683)
0927 15:17:02.497063 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:02.497067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:02.497144 (+ 77us) server_negotiation.cc:407] Connection header received
0927 15:17:02.497269 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:02.497271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:02.497307 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:02.497362 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:02.497891 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.498370 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.499138 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:02.499395 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:02.499479 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:02.501607 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:02.501610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:02.501612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:02.501637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:02.502951 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.503342 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.503345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.503347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.503420 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:02.503644 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:02.503647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:02.503649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:02.503812 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:02.503920 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:02.504151 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 15:17:02.504261 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:17:02.505393 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15835
I20260927 15:17:02.510310 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15969
I20260927 15:17:02.514910 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 16103
I20260927 15:17:02.518966 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 15741
2026-09-27T15:17:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2096 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[16252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[16252](info): set up 2 sockets
Sep 27 15:17:02 dist-test-slave-m4lt krb5kdc[16252](info): commencing operation
krb5kdc: starting...
W20260927 15:17:03.473762 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:17:03 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab.
Entry for principal oryx/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab.
I20260927 15:17:03.587889 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:35081
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab
--principal=oryx/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:35081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:03.693876 16265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:03.694159 16265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:03.694214 16265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:03.698167 16265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:03.698222 16265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:03.698247 16265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:03.698271 16265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:03.698299 16265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:03.703423 16265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:35081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab
--principal=oryx/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:35081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:03.704456 16265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:03.705446 16265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:03.713459 16270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:03.713481 16271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:03.713443 16273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:03.713743 16265 server_base.cc:1034] running on GCE node
I20260927 15:17:03.714653 16265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:03.715700 16265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:03.716881 16265 hybrid_clock.cc:648] HybridClock initialized: now 1790522223716855 us; error 38 us; skew 500 ppm
Sep 27 15:17:03 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:03.719568 16265 init.cc:377] Logged in from keytab as oryx/127.9.242.190@KRBTEST.COM (short username oryx)
I20260927 15:17:03.720412 16265 webserver.cc:460] Webserver started at http://127.9.242.190:33703/ using document root <none> and password file <none>
I20260927 15:17:03.720916 16265 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:03.720973 16265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:03.721138 16265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:03.723109 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "1f0a99a226304e7f83d3e945230fc57e"
format_stamp: "Formatted at 2026-09-27 15:17:03 on dist-test-slave-m4lt"
server_key: "be2697abe8c1eac0e827d72624ade48c"
server_key_iv: "f4555d8d9747ec13c71741f1651141c5"
server_key_version: "encryptionkey@0"
I20260927 15:17:03.723593 16265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "1f0a99a226304e7f83d3e945230fc57e"
format_stamp: "Formatted at 2026-09-27 15:17:03 on dist-test-slave-m4lt"
server_key: "be2697abe8c1eac0e827d72624ade48c"
server_key_iv: "f4555d8d9747ec13c71741f1651141c5"
server_key_version: "encryptionkey@0"
I20260927 15:17:03.727248 16265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:17:03.729612 16280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:03.731133 16265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:03.731247 16265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "1f0a99a226304e7f83d3e945230fc57e"
format_stamp: "Formatted at 2026-09-27 15:17:03 on dist-test-slave-m4lt"
server_key: "be2697abe8c1eac0e827d72624ade48c"
server_key_iv: "f4555d8d9747ec13c71741f1651141c5"
server_key_version: "encryptionkey@0"
I20260927 15:17:03.731357 16265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:03.747561 16265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:03.750658 16265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:03.750830 16265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:03.757392 16265 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:35081
I20260927 15:17:03.757634 16332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:35081 every 8 connection(s)
I20260927 15:17:03.758404 16265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:03.761344 16333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:03.763566 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 16265
I20260927 15:17:03.763669 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:17:03.763913 10186 external_mini_cluster.cc:1428] Setting key 940cbd81c2ebc0eac20dfd0c0e87cea6
I20260927 15:17:03.767303 16333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: Bootstrap starting.
I20260927 15:17:03.769850 16333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:03.770617 16333 log.cc:824] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:03.772428 16333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: No bootstrap required, opened a new log
Sep 27 15:17:03 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:03.776189 16333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } }
I20260927 15:17:03.776410 16333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:03.776461 16333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0a99a226304e7f83d3e945230fc57e, State: Initialized, Role: FOLLOWER
I20260927 15:17:03.776892 16333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } }
I20260927 15:17:03.777024 16333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:03.777087 16333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:03.777180 16333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:03.778117 16333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } }
I20260927 15:17:03.778434 16333 leader_election.cc:302] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f0a99a226304e7f83d3e945230fc57e; no voters:
I20260927 15:17:03.778707 16333 leader_election.cc:288] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:03.778803 16338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:03.778995 16338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 1 LEADER]: Becoming Leader. State: Replica: 1f0a99a226304e7f83d3e945230fc57e, State: Running, Role: LEADER
I20260927 15:17:03.779014 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:03.765270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.9.242.190:35081)
0927 15:17:03.765682 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:03.765700 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:03.765743 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:17:03.766519 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:03.766543 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:03.766951 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:03.767307 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:03.767982 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:03.769419 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:03.770327 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:03.771023 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:03.771179 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:03.773430 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:03.773436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:03.773453 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:03.773482 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:03.776003 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:03.776475 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:03.776480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:03.776485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:03.776642 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:03.776896 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:03.776899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:03.776901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:03.777363 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:03.777554 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:03.777836 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 15:17:03.778114 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260927 15:17:03.779323 16338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } }
I20260927 15:17:03.780066 16333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:03.780479 16338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f0a99a226304e7f83d3e945230fc57e. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } }
I20260927 15:17:03.780580 16338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:03.780862 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } }
I20260927 15:17:03.780952 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:03.781513 16342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:03.784819 16342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:03.790405 16342 catalog_manager.cc:1347] Generated new cluster ID: 97ede2371e7c42ca9130d39d276086d8
I20260927 15:17:03.790473 16342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:03.802594 16342 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:03.803519 16342 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:03.807699 16342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: Generated new TSK 0
I20260927 15:17:03.808303 16342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab.
Entry for principal oryx/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab.
I20260927 15:17:03.835013 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35081
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab
--principal=oryx/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:03.962314 16359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:03.962606 16359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:03.962670 16359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:03.966884 16359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:03.966952 16359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:03.967033 16359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:03.972318 16359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab
--principal=oryx/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:03.973377 16359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:03.974382 16359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:03.983636 16365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:03.983657 16364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:03.983728 16359 server_base.cc:1034] running on GCE node
W20260927 15:17:03.983657 16367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:03.984282 16359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:03.984882 16359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:03.986044 16359 hybrid_clock.cc:648] HybridClock initialized: now 1790522223986020 us; error 35 us; skew 500 ppm
Sep 27 15:17:03 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:03.988546 16359 init.cc:377] Logged in from keytab as oryx/127.9.242.129@KRBTEST.COM (short username oryx)
I20260927 15:17:03.989389 16359 webserver.cc:460] Webserver started at http://127.9.242.129:41297/ using document root <none> and password file <none>
I20260927 15:17:03.989944 16359 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:03.990008 16359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:03.990200 16359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:03.992163 16359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "8623336dd9ed46a79d3febf934f71a42"
format_stamp: "Formatted at 2026-09-27 15:17:03 on dist-test-slave-m4lt"
server_key: "21869a71dfb918753d90320f13363969"
server_key_iv: "4e43dde163f358a1414301be7f67c0f4"
server_key_version: "encryptionkey@0"
I20260927 15:17:03.992702 16359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "8623336dd9ed46a79d3febf934f71a42"
format_stamp: "Formatted at 2026-09-27 15:17:03 on dist-test-slave-m4lt"
server_key: "21869a71dfb918753d90320f13363969"
server_key_iv: "4e43dde163f358a1414301be7f67c0f4"
server_key_version: "encryptionkey@0"
I20260927 15:17:03.996335 16359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 15:17:03.998855 16374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.000149 16359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:04.000255 16359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "8623336dd9ed46a79d3febf934f71a42"
format_stamp: "Formatted at 2026-09-27 15:17:03 on dist-test-slave-m4lt"
server_key: "21869a71dfb918753d90320f13363969"
server_key_iv: "4e43dde163f358a1414301be7f67c0f4"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.000355 16359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:04.036339 16359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:04.039459 16359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:04.039644 16359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:04.040231 16359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:04.041033 16359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:04.041079 16359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.041131 16359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:04.041157 16359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.049623 16359 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:33541
I20260927 15:17:04.049727 16487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:33541 every 8 connection(s)
I20260927 15:17:04.050644 16359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:04.052009 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 16359
I20260927 15:17:04.052219 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:04.052526 10186 external_mini_cluster.cc:1428] Setting key 0bacb05bf593325f17ba1825391c1343
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.130@KRBTEST.COM" created.
I20260927 15:17:04.064702 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.052588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:33555 (local address 127.9.242.190:35081)
0927 15:17:04.052716 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.052721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.053724 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 15:17:04.055083 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.055086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.055137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.055213 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.056386 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.056916 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.058069 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.058304 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.058390 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.060924 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.060927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.060929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.060954 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.062361 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.062883 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.062886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.062887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.062974 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.063476 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.063479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.063480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.063687 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.063792 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.064448 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.064560 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:17:04.065770 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.053057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:33555)
0927 15:17:04.053543 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.053596 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.054877 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.055390 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.055398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.055755 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.056248 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.056261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.057041 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.057044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.057954 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.057963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.058435 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.058439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.058595 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.059120 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.059137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.060729 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.062498 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.062504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.062522 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.062795 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.063149 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.063153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.063155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.063346 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.063811 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.063818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.064172 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.064389 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.064832 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
I20260927 15:17:04.067183 16488 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35081
I20260927 15:17:04.067413 16488 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:04.067930 16488 heartbeater.cc:511] Master 127.9.242.190:35081 requested a full tablet report, sending...
I20260927 15:17:04.069604 16297 ts_manager.cc:194] Registered new tserver with Master: 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129:33541)
I20260927 15:17:04.070832 16297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:33555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab.
Entry for principal oryx/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab.
I20260927 15:17:04.078887 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35081
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab
--principal=oryx/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:04.188436 16493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:04.188701 16493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:04.188760 16493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:04.192848 16493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:04.192909 16493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:04.192988 16493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:04.198218 16493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab
--principal=oryx/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:04.199230 16493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:04.200310 16493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:04.209830 16499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.209844 16498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.209844 16501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:04.209899 16493 server_base.cc:1034] running on GCE node
I20260927 15:17:04.210466 16493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:04.211107 16493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:04.212291 16493 hybrid_clock.cc:648] HybridClock initialized: now 1790522224212273 us; error 31 us; skew 500 ppm
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:04.215086 16493 init.cc:377] Logged in from keytab as oryx/127.9.242.130@KRBTEST.COM (short username oryx)
I20260927 15:17:04.215943 16493 webserver.cc:460] Webserver started at http://127.9.242.130:39095/ using document root <none> and password file <none>
I20260927 15:17:04.216473 16493 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:04.216537 16493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:04.216766 16493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:04.218904 16493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "53258472abdd4851b8a46829b667eaab"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "dc10506557c6ada9c68b7d3bfb54f7ba"
server_key_iv: "409182ff7d5c8f1c2c4f94229147c652"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.219465 16493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "53258472abdd4851b8a46829b667eaab"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "dc10506557c6ada9c68b7d3bfb54f7ba"
server_key_iv: "409182ff7d5c8f1c2c4f94229147c652"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.223241 16493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:17:04.225729 16508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.227087 16493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:17:04.227188 16493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "53258472abdd4851b8a46829b667eaab"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "dc10506557c6ada9c68b7d3bfb54f7ba"
server_key_iv: "409182ff7d5c8f1c2c4f94229147c652"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.227289 16493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:04.243695 16493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:04.246461 16493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:04.246603 16493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:04.247102 16493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:04.247869 16493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:04.247916 16493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.247954 16493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:04.247968 16493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.256059 16493 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:33009
I20260927 15:17:04.256161 16621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:33009 every 8 connection(s)
I20260927 15:17:04.256913 16493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:04.265303 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 16493
I20260927 15:17:04.265420 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:04.265751 10186 external_mini_cluster.cc:1428] Setting key f63a7a4f7dec8783eca15711d17edd90
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:04.270087 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.258651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:47015 (local address 127.9.242.190:35081)
0927 15:17:04.258765 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.258768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.259563 (+ 795us) server_negotiation.cc:407] Connection header received
0927 15:17:04.260842 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.260845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.260903 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.260970 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.262196 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.262713 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.263793 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.264034 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.264123 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.266597 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.266600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.266602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.266626 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.268102 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.268539 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.268542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.268544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.268630 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.268990 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.268992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.268993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.269169 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.269240 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.269801 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.269911 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:04.271050 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.259005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:47015)
0927 15:17:04.259420 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.259461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.260639 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.261199 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.261206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.261545 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.262060 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.262072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.262822 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.262826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.263680 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.263689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.264169 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.264172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.264321 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.264824 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.264839 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.266454 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.268210 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.268215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.268225 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.268457 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.268729 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.268733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.268735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.268906 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.269251 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.269257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.269546 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.269767 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.270070 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20260927 15:17:04.272287 16622 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35081
I20260927 15:17:04.272503 16622 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:04.272969 16622 heartbeater.cc:511] Master 127.9.242.190:35081 requested a full tablet report, sending...
I20260927 15:17:04.273955 16297 ts_manager.cc:194] Registered new tserver with Master: 53258472abdd4851b8a46829b667eaab (127.9.242.130:33009)
I20260927 15:17:04.274485 16297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:47015
WARNING: no policy specified for oryx/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab.
Entry for principal oryx/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab.
I20260927 15:17:04.290531 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35081
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab
--principal=oryx/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:04.399008 16627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:04.399300 16627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:04.399362 16627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:04.403813 16627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:04.403895 16627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:04.403996 16627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:04.409453 16627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab
--principal=oryx/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:35081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:04.410538 16627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:04.411518 16627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:04.420997 16632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.421000 16633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.421000 16635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:04.421072 16627 server_base.cc:1034] running on GCE node
I20260927 15:17:04.421763 16627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:04.422374 16627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:04.423564 16627 hybrid_clock.cc:648] HybridClock initialized: now 1790522224423541 us; error 39 us; skew 500 ppm
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:04.426367 16627 init.cc:377] Logged in from keytab as oryx/127.9.242.131@KRBTEST.COM (short username oryx)
I20260927 15:17:04.427322 16627 webserver.cc:460] Webserver started at http://127.9.242.131:46001/ using document root <none> and password file <none>
I20260927 15:17:04.427860 16627 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:04.427925 16627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:04.428112 16627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:04.430174 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "8765535ee06e4c7a937d61684587108d"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "6f64c966cd077bcd16de37441af03c6c"
server_key_iv: "636532662f03e176beac70ff25be6db8"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.430732 16627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "8765535ee06e4c7a937d61684587108d"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "6f64c966cd077bcd16de37441af03c6c"
server_key_iv: "636532662f03e176beac70ff25be6db8"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.434523 16627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:17:04.437003 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.438340 16627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:04.438450 16627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "8765535ee06e4c7a937d61684587108d"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "6f64c966cd077bcd16de37441af03c6c"
server_key_iv: "636532662f03e176beac70ff25be6db8"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.438547 16627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:04.465682 16627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:04.468593 16627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:04.468771 16627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:04.469336 16627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:04.470173 16627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:04.470221 16627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.470274 16627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:04.470304 16627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.478614 16627 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:38333
I20260927 15:17:04.478659 16755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:38333 every 8 connection(s)
I20260927 15:17:04.479537 16627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:04.486654 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 16627
I20260927 15:17:04.486769 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:04.487047 10186 external_mini_cluster.cc:1428] Setting key 454ee34ce72d51e73cf41d6e30da1646
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:04.492662 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.481390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:48217 (local address 127.9.242.190:35081)
0927 15:17:04.481535 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.481539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.482465 (+ 926us) server_negotiation.cc:407] Connection header received
0927 15:17:04.483817 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.483821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.483877 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.483951 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.485144 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.485699 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.486904 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.487214 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.487319 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.489758 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.489761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.489763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.489785 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.490974 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.491419 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.491422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.491423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.491490 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.491755 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.491757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.491758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.491918 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.491985 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.492430 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.492543 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:04.493424 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.481848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:48217)
0927 15:17:04.482291 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.482354 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.483578 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.484151 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.484158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.484517 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.484995 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.485008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.485836 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.485839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.486786 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.486796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.487354 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.487357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.487535 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.488108 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.488123 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.489573 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.491069 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.491074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.491111 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.491345 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.491574 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.491575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.491577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.491692 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.491984 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.491989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.492244 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.492391 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.492630 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260927 15:17:04.494479 16756 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35081
I20260927 15:17:04.494685 16756 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:04.495105 16756 heartbeater.cc:511] Master 127.9.242.190:35081 requested a full tablet report, sending...
I20260927 15:17:04.496167 16297 ts_manager.cc:194] Registered new tserver with Master: 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:04.496714 16297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:48217
I20260927 15:17:04.500509 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.242.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.189.keytab.
Entry for principal oryx/127.9.242.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.189.keytab.
I20260927 15:17:04.524438 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:36101
--webserver_interface=127.9.242.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.189.keytab
--principal=oryx/127.9.242.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:35081,127.9.242.189:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:04.631438 16761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:04.631731 16761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:04.631789 16761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:04.635815 16761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:04.635875 16761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:04.635907 16761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:04.635927 16761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:04.635957 16761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:04.641145 16761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:35081,127.9.242.189:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.189.keytab
--principal=oryx/127.9.242.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.242.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
W20260927 15:17:04.641361 16761 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.242.190:35081,127.9.242.189:36101'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 15:17:04.642248 16761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:04.643224 16761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:04.651320 16767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.651335 16766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.651335 16769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:04.651505 16761 server_base.cc:1034] running on GCE node
I20260927 15:17:04.652241 16761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:04.653280 16761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:04.654439 16761 hybrid_clock.cc:648] HybridClock initialized: now 1790522224654406 us; error 44 us; skew 500 ppm
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:04.657117 16761 init.cc:377] Logged in from keytab as oryx/127.9.242.189@KRBTEST.COM (short username oryx)
I20260927 15:17:04.657974 16761 webserver.cc:460] Webserver started at http://127.9.242.189:44169/ using document root <none> and password file <none>
I20260927 15:17:04.658478 16761 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:04.658535 16761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:04.658713 16761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:04.660635 16761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/instance:
uuid: "fb2f05ce55dd4176a0240f3673eb6268"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "f11013b128004df4905e6f1f763518bc"
server_key_iv: "9c470d3c2148a1daa82f96f705e496ef"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.661152 16761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal/instance:
uuid: "fb2f05ce55dd4176a0240f3673eb6268"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "f11013b128004df4905e6f1f763518bc"
server_key_iv: "9c470d3c2148a1daa82f96f705e496ef"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.664651 16761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:17:04.666941 16776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.668179 16761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:04.668286 16761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
uuid: "fb2f05ce55dd4176a0240f3673eb6268"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "f11013b128004df4905e6f1f763518bc"
server_key_iv: "9c470d3c2148a1daa82f96f705e496ef"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.668375 16761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:04.688139 16761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:04.691059 16761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:04.691254 16761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:04.697610 16761 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.189:36101
I20260927 15:17:04.697878 16828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.189:36101 every 8 connection(s)
I20260927 15:17:04.698541 16761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
I20260927 15:17:04.700155 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 16761
I20260927 15:17:04.701406 16829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:04.705209 16829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.242.189@KRBTEST.COM
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.189@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:04.713531 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.701629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.9.242.189:36101)
0927 15:17:04.702045 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.702059 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.702099 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:17:04.702707 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.702722 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.703025 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.703373 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.704170 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.705144 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.706013 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.706477 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.706620 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.709087 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.709094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.709105 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.709135 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.710896 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.711352 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.711358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.711364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.711491 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.711753 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.711755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.711756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.712173 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.712363 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.712609 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.712847 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":127,"threads_started":1}
I20260927 15:17:04.715157 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.706789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.9.242.190:35081)
0927 15:17:04.706925 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.706928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.707451 (+ 523us) server_negotiation.cc:407] Connection header received
0927 15:17:04.707560 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.707564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.707616 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.707696 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.708283 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.708932 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.710102 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.710335 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.710419 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.712419 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.712421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.712424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.712446 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.713675 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.714096 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.714098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.714099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.714171 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.714406 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.714408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.714409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.714595 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.714695 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.714966 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.715049 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:17:04.715268 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.706744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:54138)
0927 15:17:04.707318 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.707363 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.707474 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.707752 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.707756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.708048 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.708170 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.708175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.709036 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.709040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.709966 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.709975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.710441 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.710444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.710540 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.710920 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.710933 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.712298 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.713758 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.713765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.713770 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.714015 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.714245 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.714247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.714249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.714331 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.714694 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.714700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.714802 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.714933 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.715098 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":334,"threads_started":1}
I20260927 15:17:04.716418 16829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.189@KRBTEST.COM for oryx/127.9.242.189@KRBTEST.COM
I20260927 15:17:04.724269 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.716835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.0.0.1:51096)
0927 15:17:04.717051 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.717064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.717145 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.717321 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.717323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.717469 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.717577 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.717582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.718717 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.718720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.719339 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.719348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.719742 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.719745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.719822 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.720188 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.720211 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.721413 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.722805 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.722808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.722810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.723051 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.723283 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.723285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.723287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.723355 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.723676 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.723679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.723760 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.723841 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.723967 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260927 15:17:04.724269 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.716951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.9.242.189:36101)
0927 15:17:04.717056 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.717060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.717108 (+ 48us) server_negotiation.cc:407] Connection header received
0927 15:17:04.717208 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.717211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.717244 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.717303 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.717700 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.718589 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.719439 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.719658 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.719738 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.721544 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.721547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.721550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.721591 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.722710 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.723136 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.723139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.723141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.723212 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.723431 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.723434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.723436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.723599 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.723676 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.723890 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.724028 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":35}
I20260927 15:17:04.727232 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Bootstrap starting.
I20260927 15:17:04.729718 16829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:04.730459 16829 log.cc:824] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:04.732409 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: No bootstrap required, opened a new log
I20260927 15:17:04.735606 16829 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } }
I20260927 15:17:04.735838 16829 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:04.735881 16829 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2f05ce55dd4176a0240f3673eb6268, State: Initialized, Role: FOLLOWER
I20260927 15:17:04.736331 16829 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } }
I20260927 15:17:04.736840 16835 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } } }
I20260927 15:17:04.736974 16835 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:17:04.737560 16829 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:17:04.742331 16846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:17:04.742426 16846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:17:04.743772 16761 master_runner.cc:428] Detected that this master fb2f05ce55dd4176a0240f3673eb6268 is joining an existing cluster
I20260927 15:17:04.743829 16761 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.242.189:36101
I20260927 15:17:04.754056 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.746067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.0.0.1:51098)
0927 15:17:04.746463 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.746477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.746598 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.746825 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.746828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.747070 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.747209 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.747216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.748161 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.748164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.749077 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.749087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.749517 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.749520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.749605 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.750092 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.750106 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.750548 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.752216 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.752219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.752222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.752449 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.752959 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.752962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.752964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.753054 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.753488 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.753490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.753567 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.753716 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.753852 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":86,"threads_started":1}
I20260927 15:17:04.754077 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.746215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.9.242.189:36101)
0927 15:17:04.746304 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.746308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.746542 (+ 234us) server_negotiation.cc:407] Connection header received
0927 15:17:04.746693 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.746695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.746729 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.746792 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.747305 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.748038 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.749204 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.749406 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.749494 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.750695 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.750699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.750701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.750721 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.752112 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.752552 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.752556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.752558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.752648 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.753147 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.753151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.753153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.753355 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.753455 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.753775 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.753919 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":27}
I20260927 15:17:04.755066 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.745743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.9.242.190:35081)
0927 15:17:04.745856 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.745860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.746461 (+ 601us) server_negotiation.cc:407] Connection header received
0927 15:17:04.746570 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.746573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.746618 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.746678 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.747221 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.748041 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.750241 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.750562 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.750664 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.751939 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.751943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.751946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.751972 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.753499 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.753943 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.753947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.753949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.754025 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.754312 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.754315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.754317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.754492 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.754571 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.754819 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.754930 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:04.755708 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.745675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:54154)
0927 15:17:04.746383 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.746393 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.746480 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.746790 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.746792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.747009 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.747115 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.747121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.749224 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.749227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.750118 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.750129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.750670 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.750674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.750765 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.751336 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.751351 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.751803 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.753595 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.753597 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.753599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.753846 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.754138 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.754140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.754142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.754218 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.754576 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.754579 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.754669 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.754742 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.755585 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":87,"threads_started":1}
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.190@KRBTEST.COM for oryx/127.9.242.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.242.188@KRBTEST.COM; defaulting to no policy
I20260927 15:17:04.776271 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.760070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.9.242.189:36101)
0927 15:17:04.760173 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.760177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.766149 (+ 5972us) server_negotiation.cc:407] Connection header received
0927 15:17:04.766410 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.766414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.766466 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.766514 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.767459 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.768223 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.769496 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.769855 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.769924 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.772863 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.772867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.772870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.772896 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.774505 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.775049 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.775052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.775053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.775134 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.775386 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.775389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.775390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.775584 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.775662 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.776008 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.776155 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:17:04.776449 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.760042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.0.0.1:51112)
0927 15:17:04.765997 (+ 5955us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.766055 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.766263 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.766687 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.766694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.767181 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.767328 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.767336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.768362 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.768365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.769368 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.769379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.770074 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.770078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.770283 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.771013 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.771027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.772712 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.774648 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.774657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.774664 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.774941 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.775212 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.775215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.775217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.775307 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.775684 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.775692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.775821 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.775972 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.776173 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5736,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.188@KRBTEST.COM" created.
I20260927 15:17:04.777966 16297 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.242.189:36101: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268"
member_type: NON_VOTER
last_known_addr {
host: "127.9.242.189"
port: 36101
}
attrs {
promote: true
}
}
dest_uuid: "1f0a99a226304e7f83d3e945230fc57e"
cas_config_opid_index: -1
I20260927 15:17:04.778658 16297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } }
I20260927 15:17:04.780400 16857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fb2f05ce55dd4176a0240f3673eb6268 (127.9.242.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } }
I20260927 15:17:04.781973 16803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:04.783164 16857 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.242.189:36101
I20260927 15:17:04.783389 16803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Refusing update from remote peer 1f0a99a226304e7f83d3e945230fc57e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 15:17:04.783555 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } }
I20260927 15:17:04.783663 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:04.784144 16761 master.cc:537] Master@127.9.242.189:36101 shutting down...
I20260927 15:17:04.784562 16858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:17:04.787137 16281 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 15:17:04.788172 16761 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:04.788493 16761 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 15:17:04.788532 16281 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e -> Peer fb2f05ce55dd4176a0240f3673eb6268 (127.9.242.189:36101): Couldn't send request to peer fb2f05ce55dd4176a0240f3673eb6268. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:17:04.788597 16761 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.188.keytab.
Entry for principal oryx/127.9.242.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.188.keytab.
I20260927 15:17:04.792762 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:36715
--webserver_interface=127.9.242.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.188.keytab
--principal=oryx/127.9.242.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:04.803071 16761 master.cc:552] Master@127.9.242.189:36101 shutdown complete.
I20260927 15:17:04.803133 16761 master_runner.cc:305] Clearing existing system tablet
I20260927 15:17:04.803366 16761 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:04.803422 16761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:04.804705 16761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 15:17:04.805518 16863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.805756 16761 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:04.806030 16761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:04.806078 16761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
uuid: "fb2f05ce55dd4176a0240f3673eb6268"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "f11013b128004df4905e6f1f763518bc"
server_key_iv: "9c470d3c2148a1daa82f96f705e496ef"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.808310 16761 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:04.810775 16761 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:17:04.810864 16761 log.cc:1196] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 15:17:04.811094 16761 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Deleting consensus metadata
I20260927 15:17:04.811249 16761 master_runner.cc:315] Copying system tablet from 127.9.242.190:35081
I20260927 15:17:04.812404 16761 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: tablet copy: Beginning tablet copy session from remote peer at address 127.9.242.190:35081
I20260927 15:17:04.821076 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.813144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.9.242.190:35081)
0927 15:17:04.813267 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:04.813271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:04.813418 (+ 147us) server_negotiation.cc:407] Connection header received
0927 15:17:04.813542 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:04.813545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:04.813595 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:04.813751 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:04.814230 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.815022 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.816179 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.816504 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.816607 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.817822 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:04.817826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:04.817829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:04.817858 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:04.819430 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.819881 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.819885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.819887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.819977 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:04.820241 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:04.820243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:04.820245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:04.820457 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:04.820553 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:04.820799 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 15:17:04.820909 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:17:04.821453 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:04.813040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:54162)
0927 15:17:04.813322 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:04.813336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:04.813453 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:04.813767 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:04.813771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:04.813976 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:04.814104 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.814111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.815136 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.815139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.816066 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:04.816077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:04.816616 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:04.816620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:04.816713 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:04.817246 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:04.817260 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:04.817695 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:04.819536 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.819539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.819542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.819785 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.820067 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:04.820070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:04.820072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:04.820158 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:04.820546 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:04.820548 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:04.820645 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:17:04.820740 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:17:04.821016 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
I20260927 15:17:04.821977 16317 tablet_copy_service.cc:140] P 1f0a99a226304e7f83d3e945230fc57e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fb2f05ce55dd4176a0240f3673eb6268 ({username='oryx', principal='oryx/127.9.242.189@KRBTEST.COM'} at 127.0.0.1:54162)
I20260927 15:17:04.822153 16317 tablet_copy_service.cc:161] P 1f0a99a226304e7f83d3e945230fc57e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fb2f05ce55dd4176a0240f3673eb6268 at {username='oryx', principal='oryx/127.9.242.189@KRBTEST.COM'} at 127.0.0.1:54162: session id = fb2f05ce55dd4176a0240f3673eb6268-00000000000000000000000000000000
I20260927 15:17:04.823843 16317 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:17:04.825485 16761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:17:04.827817 16761 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: tablet copy: Starting download of 0 data blocks...
I20260927 15:17:04.827998 16761 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: tablet copy: Starting download of 1 WAL segments...
I20260927 15:17:04.829702 16761 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:17:04.831518 16317 tablet_copy_service.cc:342] P 1f0a99a226304e7f83d3e945230fc57e: Request end of tablet copy session fb2f05ce55dd4176a0240f3673eb6268-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.242.189@KRBTEST.COM'} at 127.0.0.1:54162
I20260927 15:17:04.831668 16317 tablet_copy_service.cc:433] P 1f0a99a226304e7f83d3e945230fc57e: ending tablet copy session fb2f05ce55dd4176a0240f3673eb6268-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fb2f05ce55dd4176a0240f3673eb6268
I20260927 15:17:04.839395 16761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:04.841037 16873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.841265 16870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.841796 16871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:04.841821 16761 server_base.cc:1034] running on GCE node
I20260927 15:17:04.841964 16761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:04.842262 16761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:04.843395 16761 hybrid_clock.cc:648] HybridClock initialized: now 1790522224843385 us; error 29 us; skew 500 ppm
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:04.845103 16761 init.cc:377] Logged in from keytab as oryx/127.9.242.189@KRBTEST.COM (short username oryx)
I20260927 15:17:04.845573 16761 webserver.cc:460] Webserver started at http://127.9.242.189:34975/ using document root <none> and password file <none>
I20260927 15:17:04.845767 16761 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:04.845803 16761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:04.847067 16761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:04.848006 16880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.848234 16761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:04.848336 16761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
uuid: "fb2f05ce55dd4176a0240f3673eb6268"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "f11013b128004df4905e6f1f763518bc"
server_key_iv: "9c470d3c2148a1daa82f96f705e496ef"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.848470 16761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:04.861625 16761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:04.863821 16761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:04.868901 16761 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.189:36101
I20260927 15:17:04.868945 16932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.189:36101 every 8 connection(s)
I20260927 15:17:04.869675 16761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
I20260927 15:17:04.871524 16933 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:17:04.872565 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Bootstrap starting.
I20260927 15:17:04.882736 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:17:04.883014 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Bootstrap complete.
I20260927 15:17:04.883626 16933 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } }
I20260927 15:17:04.883761 16933 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb2f05ce55dd4176a0240f3673eb6268, State: Initialized, Role: LEARNER
I20260927 15:17:04.883864 16933 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } }
I20260927 15:17:04.884202 16936 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } }
I20260927 15:17:04.884320 16933 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:04.884400 16936 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:17:04.887710 16947 catalog_manager.cc:1259] Loaded cluster ID: 97ede2371e7c42ca9130d39d276086d8
I20260927 15:17:04.887768 16947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: loading cluster ID for follower catalog manager: success
I20260927 15:17:04.889904 16947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: acquiring CA information for follower catalog manager: success
I20260927 15:17:04.891285 16947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:17:04.921069 16861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:04.921429 16861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:04.921489 16861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:04.927047 16861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:04.927112 16861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:04.927130 16861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:04.927148 16861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:04.927176 16861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:04.932755 16861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.188.keytab
--principal=oryx/127.9.242.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:36715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.242.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:04.933911 16861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:04.934944 16861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:04.942942 16955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:04.943079 16861 server_base.cc:1034] running on GCE node
W20260927 15:17:04.942960 16952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:04.943279 16953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:04.943770 16861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:04.944850 16861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:04.945998 16861 hybrid_clock.cc:648] HybridClock initialized: now 1790522224945984 us; error 28 us; skew 500 ppm
Sep 27 15:17:04 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:04.948529 16861 init.cc:377] Logged in from keytab as oryx/127.9.242.188@KRBTEST.COM (short username oryx)
I20260927 15:17:04.949417 16861 webserver.cc:460] Webserver started at http://127.9.242.188:45419/ using document root <none> and password file <none>
I20260927 15:17:04.950055 16861 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:04.950119 16861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:04.950314 16861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:04.952271 16861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/instance:
uuid: "fbe07224a450408e89e7487300022d12"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "05ca85c1e5c943f99fe92256fa9db633"
server_key_iv: "2932762dd6b044fcacbb480c266ddd11"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.952802 16861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal/instance:
uuid: "fbe07224a450408e89e7487300022d12"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "05ca85c1e5c943f99fe92256fa9db633"
server_key_iv: "2932762dd6b044fcacbb480c266ddd11"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.956307 16861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:17:04.958680 16962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:04.960026 16861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:04.960131 16861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
uuid: "fbe07224a450408e89e7487300022d12"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "05ca85c1e5c943f99fe92256fa9db633"
server_key_iv: "2932762dd6b044fcacbb480c266ddd11"
server_key_version: "encryptionkey@0"
I20260927 15:17:04.960215 16861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:04.992008 16861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:04.995072 16861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:04.995235 16861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:05.001782 16861 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.188:36715
I20260927 15:17:05.002074 17014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.188:36715 every 8 connection(s)
I20260927 15:17:05.003013 16861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
I20260927 15:17:05.005990 17015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:05.009447 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 16861
I20260927 15:17:05.009871 17015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.188@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.242.188@KRBTEST.COM
I20260927 15:17:05.022284 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.011775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.9.242.190:35081)
0927 15:17:05.011895 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.011899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.012189 (+ 290us) server_negotiation.cc:407] Connection header received
0927 15:17:05.012715 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.012718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.012763 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.012830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.013815 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.014342 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.015440 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.015688 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.015762 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.018503 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.018506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.018509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.018535 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.020000 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.020614 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.020616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.020617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.020697 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.021043 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.021047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.021049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.021346 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.021436 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.021978 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.022114 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:17:05.023195 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.011647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:54176)
0927 15:17:05.012037 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.012089 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.012502 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.012903 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.012908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.013348 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.013623 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.013635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.014464 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.014467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.015318 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.015326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.015833 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.015837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.016026 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.016693 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.016711 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.018320 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.020104 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.020111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.020123 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.020508 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.020800 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.020803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.020805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.020953 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.021445 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.021452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.021761 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.021922 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.022268 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":43,"thread_start_us":135,"threads_started":1}
I20260927 15:17:05.023200 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.010880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.9.242.188:36715)
0927 15:17:05.011325 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.011344 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.011488 (+ 144us) server_negotiation.cc:407] Connection header received
0927 15:17:05.012497 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.012521 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.012758 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.012937 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.013626 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.014699 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.015517 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.015975 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.016064 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.018861 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.018866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.018879 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.018900 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.020621 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.021010 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.021017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.021019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.021090 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.021381 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.021383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.021384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.021783 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.021983 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.022161 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.022322 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260927 15:17:05.024432 17015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.188@KRBTEST.COM for oryx/127.9.242.189@KRBTEST.COM
I20260927 15:17:05.032584 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.024911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.0.0.1:51128)
0927 15:17:05.025241 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.025254 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.025328 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.025690 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.025692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.025855 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.025952 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.025957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.026863 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.026865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.027432 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.027439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.027884 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.027886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.027957 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.028441 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.028452 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.029729 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.031340 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.031342 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.031344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.031544 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.031784 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.031786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.031787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.031856 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.032160 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.032162 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.032239 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.032308 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.032461 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260927 15:17:05.032591 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.025008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.9.242.189:36101)
0927 15:17:05.025266 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.025271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.025287 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:17:05.025419 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.025422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.025473 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.025551 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.026055 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.026717 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.027539 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.027790 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.027871 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.029861 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.029865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.029868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.029897 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.031245 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.031628 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.031631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.031632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.031712 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.031919 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.031921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.031922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.032097 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.032166 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.032348 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.032448 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260927 15:17:05.033336 17015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522224, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.188@KRBTEST.COM for oryx/127.9.242.188@KRBTEST.COM
I20260927 15:17:05.040367 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.033828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.9.242.188:36715)
0927 15:17:05.033916 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.033920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.033958 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:17:05.034088 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.034097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.034145 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.034208 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.034532 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.035147 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.035826 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.036093 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.036177 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.037893 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.037896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.037899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.037920 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.039034 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.039367 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.039370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.039372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.039436 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.039646 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.039649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.039651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.039804 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.039880 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.040085 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.040222 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":38}
I20260927 15:17:05.040367 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.033808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.0.0.1:50022)
0927 15:17:05.033905 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.033918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.034005 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.034227 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.034229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.034374 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.034437 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.034441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.035246 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.035249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.035734 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.035741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.036183 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.036186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.036253 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.036610 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.036620 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.037783 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.039126 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.039129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.039131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.039294 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.039506 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.039509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.039510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.039576 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.039879 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.039881 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.039956 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.040036 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.040173 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 15:17:05.043694 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Bootstrap starting.
I20260927 15:17:05.045943 17015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:05.046638 17015 log.cc:824] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:05.048556 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: No bootstrap required, opened a new log
I20260927 15:17:05.051602 17015 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } }
I20260927 15:17:05.051795 17015 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:05.051826 17015 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe07224a450408e89e7487300022d12, State: Initialized, Role: FOLLOWER
I20260927 15:17:05.052254 17015 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } }
I20260927 15:17:05.052702 17021 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } } }
I20260927 15:17:05.052845 17021 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:17:05.053478 17015 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 15:17:05.059060 17032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:17:05.059154 17032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:17:05.061686 16861 master_runner.cc:428] Detected that this master fbe07224a450408e89e7487300022d12 is joining an existing cluster
I20260927 15:17:05.061745 16861 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.242.188:36715
I20260927 15:17:05.072538 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.064610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.9.242.189:36101)
0927 15:17:05.065019 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.065024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.065226 (+ 202us) server_negotiation.cc:407] Connection header received
0927 15:17:05.065390 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.065394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.065441 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.065485 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.066025 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.066792 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.067830 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.068133 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.068193 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.069446 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.069449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.069451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.069474 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.070895 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.071366 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.071370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.071372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.071468 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.071731 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.071734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.071736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.071940 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.072019 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.072284 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.072392 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329}
I20260927 15:17:05.072552 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.064224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.0.0.1:51140)
0927 15:17:05.065154 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.065167 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.065289 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.065606 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.065608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.065808 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.065903 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.065909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.066912 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.066915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.067726 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.067732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.068338 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.068342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.068419 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.068808 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.068821 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.069142 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.071012 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.071015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.071017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.071260 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.071548 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.071551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.071553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.071646 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.072023 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.072026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.072132 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.072225 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.072358 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260927 15:17:05.073235 16488 heartbeater.cc:503] Master 127.9.242.190:35081 was elected leader, sending a full tablet report...
I20260927 15:17:05.073604 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.064591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.9.242.190:35081)
0927 15:17:05.064704 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.064707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.064720 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:05.064755 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.064758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.064797 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.064870 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.065486 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.066320 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.068450 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.068762 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.068833 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.070034 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.070038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.070041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.070070 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.071775 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.072193 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.072196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.072198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.072280 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.072551 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.072554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.072556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.072747 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.072811 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.073335 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.073450 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:17:05.074173 17039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.063805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:54190)
0927 15:17:05.064296 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.064312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.064402 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.064903 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.064906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.065202 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.065343 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.065351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.066457 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.066460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.068325 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.068335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.068973 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.068977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.069066 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.069439 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.069453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.069888 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.071859 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.071862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.071864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.072073 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.072378 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.072381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.072383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.072461 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.073052 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.073055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.073152 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.073843 (+ 691us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.073983 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":27,"spinlock_wait_cycles":768,"thread_start_us":98,"threads_started":1}
I20260927 15:17:05.076794 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 16359
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522223, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.190@KRBTEST.COM for oryx/127.9.242.188@KRBTEST.COM
I20260927 15:17:05.083151 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:33541
--local_ip_for_outbound_sockets=127.9.242.129
--tserver_master_addrs=127.9.242.190:35081
--webserver_port=41297
--webserver_interface=127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab
--principal=oryx/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:05.089746 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.078250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.0.0.1:50038)
0927 15:17:05.078417 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.078433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.078527 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.079279 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.079281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.079453 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.079540 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.079546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.080356 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.080359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.080994 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.081003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.081431 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.081434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.081527 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.082021 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.082033 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.083421 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.088116 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.088122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.088371 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.088669 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.088674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.088754 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.089206 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.089209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.089307 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.089382 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.089538 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 15:17:05.089811 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.063899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.0.0.1:50036)
0927 15:17:05.064462 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.064474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.064604 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.064888 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.064890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.065187 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.065312 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.065319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.066224 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.066227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.067099 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.067106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.067740 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.067742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.067821 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.068323 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.068337 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.085983 (+ 17646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.088251 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.088257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.088478 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.088745 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.088750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.088823 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.089250 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.089252 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.089340 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.089432 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.089571 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260927 15:17:05.090291 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.063878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.9.242.188:36715)
0927 15:17:05.063971 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.063976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.064532 (+ 556us) server_negotiation.cc:407] Connection header received
0927 15:17:05.064690 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.064693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.064730 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.064771 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.065403 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.066128 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.067195 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.067556 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.067637 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.086118 (+ 18481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.086122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.086124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.086146 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.088137 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088572 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.088575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.088588 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.088659 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088906 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.088908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.088910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.089128 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.089217 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.090058 (+ 841us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.090156 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":29}
I20260927 15:17:05.090430 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.078303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50038 (local address 127.9.242.188:36715)
0927 15:17:05.079043 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.079049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.079060 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:17:05.079124 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.079127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.079168 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.079242 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.079648 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.080261 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.081099 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.081337 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.081419 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.083584 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.083588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.083591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.083616 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.087983 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088481 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.088484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.088486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.088575 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.088850 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.088853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.088854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.089091 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.090029 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.090123 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.090234 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"spinlock_wait_cycles":35584,"thread_start_us":596,"threads_started":1}
I20260927 15:17:05.091477 16297 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.242.188:36715: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fbe07224a450408e89e7487300022d12"
member_type: NON_VOTER
last_known_addr {
host: "127.9.242.188"
port: 36715
}
attrs {
promote: true
}
}
dest_uuid: "1f0a99a226304e7f83d3e945230fc57e"
cas_config_opid_index: 5
I20260927 15:17:05.092204 16297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } }
I20260927 15:17:05.093321 16857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fbe07224a450408e89e7487300022d12 (127.9.242.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.094695 16857 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.242.188:36715
I20260927 15:17:05.094868 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.094980 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:05.095289 16861 master.cc:537] Master@127.9.242.188:36715 shutting down...
I20260927 15:17:05.095422 16989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:05.096947 16989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Refusing update from remote peer 1f0a99a226304e7f83d3e945230fc57e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:17:05.097620 16858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 15:17:05.099305 16282 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e -> Peer fbe07224a450408e89e7487300022d12 (127.9.242.188:36715): Couldn't send request to peer fbe07224a450408e89e7487300022d12. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:17:05.099545 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.093524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.0.0.1:51144)
0927 15:17:05.095312 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.095326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.095430 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.095856 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.095859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.095876 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:05.095960 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.095965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.096936 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.096940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.098083 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.098092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.099145 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.099149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.099237 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.099284 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.099351 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.099420 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1723}
I20260927 15:17:05.099949 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.093830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.9.242.189:36101)
0927 15:17:05.093945 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.093950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.095518 (+ 1568us) server_negotiation.cc:407] Connection header received
0927 15:17:05.095574 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.095577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.095641 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.095734 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:05.096075 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.096812 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.098226 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.098823 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.098916 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.099075 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.099767 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.099839 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:17:05.100504 16907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 LEARNER]: Refusing update from remote peer 1f0a99a226304e7f83d3e945230fc57e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 15:17:05.100896 16858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 15:17:05.101089 16861 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:05.101366 16861 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:05.101459 16861 tablet_replica.cc:330] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: stopping tablet replica
I20260927 15:17:05.102800 16907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fbe07224a450408e89e7487300022d12 (127.9.242.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.104523 16936 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.104635 16936 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:17:05.104558 17044 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f0a99a226304e7f83d3e945230fc57e. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: NON_VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: true } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.104676 17044 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:17:05.106184 16858 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: attempting to promote NON_VOTER fb2f05ce55dd4176a0240f3673eb6268 to VOTER
I20260927 15:17:05.106700 16858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } }
I20260927 15:17:05.107843 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.107571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.0.0.1:50044)
0927 15:17:05.107710 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.107772 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 15:17:05.108275 16907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 LEARNER]: Refusing update from remote peer 1f0a99a226304e7f83d3e945230fc57e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 15:17:05.108603 16854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.108388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.0.0.1:50046)
0927 15:17:05.108503 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.108544 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:17:05.108687 16857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260927 15:17:05.109009 16282 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e -> Peer fbe07224a450408e89e7487300022d12 (127.9.242.188:36715): Couldn't send request to peer fbe07224a450408e89e7487300022d12. Status: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36715: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:17:05.110214 16857 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fb2f05ce55dd4176a0240f3673eb6268 (127.9.242.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.111599 16857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.111698 16857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:05.111869 17045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.112072 17045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:05.112198 16907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fb2f05ce55dd4176a0240f3673eb6268 (127.9.242.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.113847 17044 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.113958 17044 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:17:05.118361 16861 master.cc:552] Master@127.9.242.188:36715 shutdown complete.
I20260927 15:17:05.118415 16861 master_runner.cc:305] Clearing existing system tablet
I20260927 15:17:05.118642 16861 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:05.118695 16861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:05.119910 16861 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 15:17:05.120707 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.120934 16861 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:05.121052 16861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:05.121080 16861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
uuid: "fbe07224a450408e89e7487300022d12"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "05ca85c1e5c943f99fe92256fa9db633"
server_key_iv: "2932762dd6b044fcacbb480c266ddd11"
server_key_version: "encryptionkey@0"
I20260927 15:17:05.123018 16861 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:05.125329 16861 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 15:17:05.125414 16861 log.cc:1196] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 15:17:05.125670 16861 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Deleting consensus metadata
I20260927 15:17:05.125819 16861 master_runner.cc:315] Copying system tablet from 127.9.242.190:35081
I20260927 15:17:05.126909 16861 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: tablet copy: Beginning tablet copy session from remote peer at address 127.9.242.190:35081
I20260927 15:17:05.135583 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.127446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:54198)
0927 15:17:05.127730 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.127744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.127856 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.128115 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.128118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.128323 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.128446 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.128452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.129442 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.129446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.130395 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.130406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.130998 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.131002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.131102 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.131649 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.131663 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.132128 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.134004 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.134007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.134009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.134257 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.134537 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.134540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.134542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.134627 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.135023 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.135026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.135133 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.135244 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.135380 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260927 15:17:05.135665 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.127551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.9.242.190:35081)
0927 15:17:05.127684 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.127689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.127821 (+ 132us) server_negotiation.cc:407] Connection header received
0927 15:17:05.127935 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.127938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.127989 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.128086 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.128560 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.129333 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.130529 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.130875 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.130973 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.132292 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.132296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.132299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.132328 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.133906 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.134361 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.134365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.134367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.134456 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.134720 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.134723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.134724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.134934 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.135024 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.135389 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.135508 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:17:05.136445 16317 tablet_copy_service.cc:140] P 1f0a99a226304e7f83d3e945230fc57e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fbe07224a450408e89e7487300022d12 ({username='oryx', principal='oryx/127.9.242.188@KRBTEST.COM'} at 127.0.0.1:54198)
I20260927 15:17:05.136545 16317 tablet_copy_service.cc:161] P 1f0a99a226304e7f83d3e945230fc57e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fbe07224a450408e89e7487300022d12 at {username='oryx', principal='oryx/127.9.242.188@KRBTEST.COM'} at 127.0.0.1:54198: session id = fbe07224a450408e89e7487300022d12-00000000000000000000000000000000
I20260927 15:17:05.137568 16317 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:17:05.138823 16861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 15:17:05.141130 16861 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: tablet copy: Starting download of 0 data blocks...
I20260927 15:17:05.141794 16861 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: tablet copy: Starting download of 1 WAL segments...
I20260927 15:17:05.143275 16861 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:17:05.145125 16317 tablet_copy_service.cc:342] P 1f0a99a226304e7f83d3e945230fc57e: Request end of tablet copy session fbe07224a450408e89e7487300022d12-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.242.188@KRBTEST.COM'} at 127.0.0.1:54198
I20260927 15:17:05.145208 16317 tablet_copy_service.cc:433] P 1f0a99a226304e7f83d3e945230fc57e: ending tablet copy session fbe07224a450408e89e7487300022d12-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fbe07224a450408e89e7487300022d12
I20260927 15:17:05.151943 16861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:05.153594 17057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:05.153846 17054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:05.154141 17055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:05.154340 16861 server_base.cc:1034] running on GCE node
I20260927 15:17:05.154488 16861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:05.154793 16861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:05.155928 16861 hybrid_clock.cc:648] HybridClock initialized: now 1790522225155913 us; error 35 us; skew 500 ppm
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:05.157413 16861 init.cc:377] Logged in from keytab as oryx/127.9.242.188@KRBTEST.COM (short username oryx)
I20260927 15:17:05.157857 16861 webserver.cc:460] Webserver started at http://127.9.242.188:46829/ using document root <none> and password file <none>
I20260927 15:17:05.158025 16861 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:05.158063 16861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:05.159252 16861 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:05.160137 17064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.160385 16861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:05.160454 16861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
uuid: "fbe07224a450408e89e7487300022d12"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "05ca85c1e5c943f99fe92256fa9db633"
server_key_iv: "2932762dd6b044fcacbb480c266ddd11"
server_key_version: "encryptionkey@0"
I20260927 15:17:05.160529 16861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:05.203110 16861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:05.205170 16861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:05.209900 16861 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.188:36715
I20260927 15:17:05.210583 16861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
I20260927 15:17:05.212553 17117 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:17:05.213578 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Bootstrap starting.
I20260927 15:17:05.217681 17116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.188:36715 every 8 connection(s)
I20260927 15:17:05.224470 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:17:05.224748 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: Bootstrap complete.
I20260927 15:17:05.225251 17117 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } }
I20260927 15:17:05.225368 17117 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fbe07224a450408e89e7487300022d12, State: Initialized, Role: LEARNER
I20260927 15:17:05.225457 17117 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } }
I20260927 15:17:05.225821 17123 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.225891 17117 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:05.225950 17123 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: This master's current role is: LEARNER
W20260927 15:17:05.226290 17042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:05.226533 17042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:05.226599 17042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:17:05.228652 17134 catalog_manager.cc:1259] Loaded cluster ID: 97ede2371e7c42ca9130d39d276086d8
I20260927 15:17:05.228708 17134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: loading cluster ID for follower catalog manager: success
I20260927 15:17:05.230890 17134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: acquiring CA information for follower catalog manager: success
I20260927 15:17:05.232224 17134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 15:17:05.232434 17042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:05.232482 17042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:05.232544 17042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:05.238047 17042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab
--principal=oryx/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:33541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=41297
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:05.239106 17042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:05.240070 17042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:05.249452 17139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:05.249469 17137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:05.249452 17136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:05.250566 17042 server_base.cc:1034] running on GCE node
I20260927 15:17:05.251004 17042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:05.251745 17042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:05.252930 17042 hybrid_clock.cc:648] HybridClock initialized: now 1790522225252903 us; error 41 us; skew 500 ppm
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:05.255970 17042 init.cc:377] Logged in from keytab as oryx/127.9.242.129@KRBTEST.COM (short username oryx)
I20260927 15:17:05.257115 17042 webserver.cc:460] Webserver started at http://127.9.242.129:41297/ using document root <none> and password file <none>
I20260927 15:17:05.257822 17042 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:05.257925 17042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:05.262403 17042 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:17:05.264333 17146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.265514 17042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:05.265607 17042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "8623336dd9ed46a79d3febf934f71a42"
format_stamp: "Formatted at 2026-09-27 15:17:03 on dist-test-slave-m4lt"
server_key: "21869a71dfb918753d90320f13363969"
server_key_iv: "4e43dde163f358a1414301be7f67c0f4"
server_key_version: "encryptionkey@0"
I20260927 15:17:05.266175 17042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:05.276479 16622 heartbeater.cc:503] Master 127.9.242.190:35081 was elected leader, sending a full tablet report...
I20260927 15:17:05.296998 17042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:05.300098 17042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:05.300246 17042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:05.300771 17042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:05.301568 17042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:05.301611 17042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.301673 17042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:05.301703 17042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.310122 17042 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:33541
I20260927 15:17:05.310168 17259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:33541 every 8 connection(s)
I20260927 15:17:05.311079 17042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:05.319944 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 17042
I20260927 15:17:05.320148 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 16493
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.189@KRBTEST.COM
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.188@KRBTEST.COM
I20260927 15:17:05.327785 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:33009
--local_ip_for_outbound_sockets=127.9.242.130
--tserver_master_addrs=127.9.242.190:35081
--webserver_port=39095
--webserver_interface=127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab
--principal=oryx/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:05.328141 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.314594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:52795 (local address 127.9.242.189:36101)
0927 15:17:05.314709 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.314713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.314867 (+ 154us) server_negotiation.cc:407] Connection header received
0927 15:17:05.315099 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.315102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.315156 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.315227 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.316524 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.317365 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319187 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.319427 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319485 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.323788 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.323792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.323794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.323823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.325574 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.326235 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.326240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.326242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.326332 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.326820 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.326825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.326827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.327058 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.327125 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.327867 (+ 742us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.327992 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:17:05.328173 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.312909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:38075 (local address 127.9.242.188:36715)
0927 15:17:05.313137 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.313142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.314095 (+ 953us) server_negotiation.cc:407] Connection header received
0927 15:17:05.315150 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.315154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.315205 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.315261 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.317488 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.318184 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319153 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.319513 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319610 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.323776 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.323780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.323783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.323812 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.325471 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.326239 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.326243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.326245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.326335 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.326752 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.326755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.326757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.327026 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.327109 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.327862 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.328002 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260927 15:17:05.328855 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.313182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:57405 (local address 127.9.242.190:35081)
0927 15:17:05.313294 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.313297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.314103 (+ 806us) server_negotiation.cc:407] Connection header received
0927 15:17:05.315151 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.315154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.315205 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.315258 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.316560 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.317355 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319193 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.319521 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319609 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.323284 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.323288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.323291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.323324 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.326669 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.327200 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.327203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.327205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.327298 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.327589 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.327592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.327594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.327813 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.327886 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.328178 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.328700 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:17:05.329736 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.314489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.9.242.129:52795)
0927 15:17:05.314785 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.314798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.314923 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.315411 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.315418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.315822 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.316390 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.316405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.317506 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.317510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.319055 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.319063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.319661 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.319927 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.320652 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.320666 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.323628 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.325771 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.325774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.325777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.326106 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.326486 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.326488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.326490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.326649 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.327217 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.327220 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.327611 (+ 391us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.328297 (+ 686us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.328671 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":158,"spinlock_wait_cycles":134912,"thread_start_us":64,"threads_started":1}
I20260927 15:17:05.329752 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.313434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:57405)
0927 15:17:05.313933 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.313992 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.314929 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.315412 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.315419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.315861 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.316398 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.316407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.317506 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.317510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.319052 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.319063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.320195 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.320203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.320312 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.320903 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.320917 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.323094 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.326791 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.326794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.326796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.327099 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.327397 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.327400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.327402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.327477 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.327935 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.327938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.328014 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.329432 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.329568 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":211,"spinlock_wait_cycles":11904,"thread_start_us":139,"threads_started":1}
I20260927 15:17:05.330356 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.313434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.9.242.129:38075)
0927 15:17:05.313933 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.313992 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.314923 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.315412 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.315419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.315833 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.316401 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.316407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.318296 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.318299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.319045 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.319054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.319752 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.319754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.319926 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.320538 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.320559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.323621 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.325613 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.325620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.325642 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.326108 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.326452 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.326455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.326457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.326644 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.327171 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.327178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.327577 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.328590 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.328725 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":194,"spinlock_wait_cycles":48896,"thread_start_us":170,"threads_started":1}
I20260927 15:17:05.331082 17260 heartbeater.cc:347] Connected to a master server at 127.9.242.188:36715
I20260927 15:17:05.331142 17261 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35081
I20260927 15:17:05.331316 17261 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.331338 17262 heartbeater.cc:347] Connected to a master server at 127.9.242.189:36101
I20260927 15:17:05.331409 17262 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.331315 17260 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.331912 17261 heartbeater.cc:511] Master 127.9.242.190:35081 requested a full tablet report, sending...
I20260927 15:17:05.331911 17260 heartbeater.cc:511] Master 127.9.242.188:36715 requested a full tablet report, sending...
I20260927 15:17:05.331912 17262 heartbeater.cc:511] Master 127.9.242.189:36101 requested a full tablet report, sending...
I20260927 15:17:05.333498 16897 ts_manager.cc:194] Registered new tserver with Master: 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129:33541)
I20260927 15:17:05.333498 17081 ts_manager.cc:194] Registered new tserver with Master: 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129:33541)
I20260927 15:17:05.334084 16297 ts_manager.cc:194] Re-registered known tserver with Master: 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129:33541)
I20260927 15:17:05.334844 16297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:57405
W20260927 15:17:05.441139 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:05.441418 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:05.441478 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:05.445614 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:05.445694 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:05.445767 17268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:05.451049 17268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab
--principal=oryx/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:33009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=39095
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:05.452095 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:05.453118 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:05.462738 17274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:05.462778 17273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:05.462767 17276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:05.462990 17268 server_base.cc:1034] running on GCE node
I20260927 15:17:05.463467 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:05.464071 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:05.465224 17268 hybrid_clock.cc:648] HybridClock initialized: now 1790522225465198 us; error 34 us; skew 500 ppm
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:05.467882 17268 init.cc:377] Logged in from keytab as oryx/127.9.242.130@KRBTEST.COM (short username oryx)
I20260927 15:17:05.468781 17268 webserver.cc:460] Webserver started at http://127.9.242.130:39095/ using document root <none> and password file <none>
I20260927 15:17:05.469347 17268 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:05.469425 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:05.473274 17268 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:05.475302 17283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.476503 17268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:05.476609 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "53258472abdd4851b8a46829b667eaab"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "dc10506557c6ada9c68b7d3bfb54f7ba"
server_key_iv: "409182ff7d5c8f1c2c4f94229147c652"
server_key_version: "encryptionkey@0"
I20260927 15:17:05.477138 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:05.498672 16756 heartbeater.cc:503] Master 127.9.242.190:35081 was elected leader, sending a full tablet report...
I20260927 15:17:05.498907 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:05.501945 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:05.502091 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:05.502615 17268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:05.503417 17268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:05.503468 17268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.503505 17268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:05.503525 17268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.512364 17268 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:33009
I20260927 15:17:05.512393 17396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:33009 every 8 connection(s)
I20260927 15:17:05.513362 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:05.515923 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 17268
I20260927 15:17:05.516203 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 16627
I20260927 15:17:05.522639 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:38333
--local_ip_for_outbound_sockets=127.9.242.131
--tserver_master_addrs=127.9.242.190:35081
--webserver_port=46001
--webserver_interface=127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab
--principal=oryx/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.189@KRBTEST.COM
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.188@KRBTEST.COM
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:05.530332 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.515114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:34789 (local address 127.9.242.188:36715)
0927 15:17:05.515242 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.515247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.516368 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 15:17:05.517523 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.517527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.517581 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.517692 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.519163 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.519933 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522025 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.522398 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522462 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.525928 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.525934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.525938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.525967 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.527873 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.528591 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.528595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.528597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.528675 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.529047 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.529050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.529052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.529257 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.529334 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.530045 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.530170 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:17:05.530332 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.517195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:57053 (local address 127.9.242.190:35081)
0927 15:17:05.517307 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.517311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.518867 (+ 1556us) server_negotiation.cc:407] Connection header received
0927 15:17:05.519053 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.519055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.519105 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.519154 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.519685 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.520384 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522001 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.522329 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522399 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.526455 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.526459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.526462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.526487 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.528188 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.528632 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.528635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.528637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.528711 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.529047 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.529050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.529052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.529257 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.529334 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.530045 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.530170 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:17:05.531414 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.515656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.9.242.130:34789)
0927 15:17:05.516189 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.516248 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.517295 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.517863 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.517872 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.518334 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.519015 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.519032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.520092 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.520095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.521862 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.521872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.522596 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.522724 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.523344 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.523363 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.525747 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.527992 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.527995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.527997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.528471 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.528775 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.528778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.528780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.528949 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.529397 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.529404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.529755 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.530517 (+ 762us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.530663 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":279,"spinlock_wait_cycles":10368,"thread_start_us":136,"threads_started":1}
I20260927 15:17:05.531658 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.515676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.9.242.130:58923)
0927 15:17:05.518486 (+ 2810us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.518501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.518595 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.518921 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.518924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.519117 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.519225 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.519232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.520278 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.520281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.521840 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.521850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.522526 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.522715 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.523588 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.523602 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.525342 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.527961 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.527967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.527985 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.528490 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.528815 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.528818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.528820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.528950 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.529435 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.529438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.529787 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.529981 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.530347 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2744,"mutex_wait_us":185,"spinlock_wait_cycles":83456,"thread_start_us":117,"threads_started":1}
I20260927 15:17:05.531425 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.516702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:57053)
0927 15:17:05.518789 (+ 2087us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.518801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.518926 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.519283 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.519285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.519459 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.519562 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.519568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.520523 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.520526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.521862 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.521872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.522559 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.523031 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.523450 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.523463 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.526306 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.528298 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.528301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.528303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.528539 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.528799 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.528802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.528803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.528945 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.529397 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.529404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.529755 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.530785 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.530897 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"mutex_wait_us":225,"spinlock_wait_cycles":5504,"thread_start_us":69,"threads_started":1}
I20260927 15:17:05.532240 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.515257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:58923 (local address 127.9.242.189:36101)
0927 15:17:05.515351 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.515354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.518599 (+ 3245us) server_negotiation.cc:407] Connection header received
0927 15:17:05.518677 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.518680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.518737 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.518808 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.519359 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.520146 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.521960 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.522256 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.522351 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.525501 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.525505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.525508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.525537 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.527827 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.528604 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.528608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.528610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.528696 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.529071 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.529074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.529076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.529297 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.529390 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.531976 (+ 2586us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.532092 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 15:17:05.533069 17398 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35081
I20260927 15:17:05.533120 17399 heartbeater.cc:347] Connected to a master server at 127.9.242.189:36101
I20260927 15:17:05.533315 17399 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.533344 17397 heartbeater.cc:347] Connected to a master server at 127.9.242.188:36715
I20260927 15:17:05.533416 17397 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.533316 17398 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.533915 17397 heartbeater.cc:511] Master 127.9.242.188:36715 requested a full tablet report, sending...
I20260927 15:17:05.533914 17398 heartbeater.cc:511] Master 127.9.242.190:35081 requested a full tablet report, sending...
I20260927 15:17:05.533914 17399 heartbeater.cc:511] Master 127.9.242.189:36101 requested a full tablet report, sending...
I20260927 15:17:05.535053 16297 ts_manager.cc:194] Re-registered known tserver with Master: 53258472abdd4851b8a46829b667eaab (127.9.242.130:33009)
I20260927 15:17:05.535089 16897 ts_manager.cc:194] Registered new tserver with Master: 53258472abdd4851b8a46829b667eaab (127.9.242.130:33009)
I20260927 15:17:05.535503 17081 ts_manager.cc:194] Registered new tserver with Master: 53258472abdd4851b8a46829b667eaab (127.9.242.130:33009)
I20260927 15:17:05.535797 16297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:57053
W20260927 15:17:05.634318 17404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:05.634644 17404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:05.634696 17404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:05.638707 17404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:05.638769 17404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:05.638839 17404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:05.644132 17404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab
--principal=oryx/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:38333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=46001
--tserver_master_addrs=127.9.242.190:35081,127.9.242.189:36101,127.9.242.188:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:05.645200 17404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:05.646281 17404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:05.656087 17409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:05.656046 17412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:05.656209 17404 server_base.cc:1034] running on GCE node
W20260927 15:17:05.656051 17410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:05.656803 17404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:05.657401 17404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:05.658564 17404 hybrid_clock.cc:648] HybridClock initialized: now 1790522225658519 us; error 56 us; skew 500 ppm
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:05.661176 17404 init.cc:377] Logged in from keytab as oryx/127.9.242.131@KRBTEST.COM (short username oryx)
I20260927 15:17:05.662071 17404 webserver.cc:460] Webserver started at http://127.9.242.131:46001/ using document root <none> and password file <none>
I20260927 15:17:05.662586 17404 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:05.662652 17404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:05.666415 17404 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:05.668323 17420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.669613 17404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:05.669744 17404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "8765535ee06e4c7a937d61684587108d"
format_stamp: "Formatted at 2026-09-27 15:17:04 on dist-test-slave-m4lt"
server_key: "6f64c966cd077bcd16de37441af03c6c"
server_key_iv: "636532662f03e176beac70ff25be6db8"
server_key_version: "encryptionkey@0"
I20260927 15:17:05.670266 17404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:05.680827 17404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:05.683773 17404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:05.683938 17404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:05.684484 17404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:05.685319 17404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:05.685364 17404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.685401 17404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:05.685420 17404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:05.693817 17404 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:38333
I20260927 15:17:05.693856 17533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:38333 every 8 connection(s)
I20260927 15:17:05.694772 17404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:05.699393 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 17404
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.189@KRBTEST.COM
Sep 27 15:17:05 dist-test-slave-m4lt krb5kdc[16252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522225, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.188@KRBTEST.COM
I20260927 15:17:05.711005 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.698295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:49955 (local address 127.9.242.189:36101)
0927 15:17:05.698413 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.698418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.698496 (+ 78us) server_negotiation.cc:407] Connection header received
0927 15:17:05.698855 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.698858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.698911 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.698977 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.700654 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.701286 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.703354 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.703655 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.703722 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.707073 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.707077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.707090 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.707119 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.708661 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.709257 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.709261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.709263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.709344 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.709756 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.709760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.709762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.709973 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.710042 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.710734 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.710839 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:17:05.712672 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.697039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:59067)
0927 15:17:05.697939 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.697952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.698646 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.699153 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.699162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.699745 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.700492 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.700511 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.701558 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.701562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.703094 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.703105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.703687 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.703690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.703875 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.704508 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.704529 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.707242 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.710911 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.710914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.710917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.711409 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.711692 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.711695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.711697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.711789 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.712201 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.712203 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.712299 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.712383 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.712505 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":491,"spinlock_wait_cycles":75392,"thread_start_us":173,"threads_started":1}
I20260927 15:17:05.713352 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.696499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:59067 (local address 127.9.242.190:35081)
0927 15:17:05.696638 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.696641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.698024 (+ 1383us) server_negotiation.cc:407] Connection header received
0927 15:17:05.698873 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.698877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.698930 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.698981 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.700631 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.701449 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.703234 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.703542 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.703644 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.707392 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.707396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.707399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.707426 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.710804 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.711519 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.711522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.711523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.711604 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.711887 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.711891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.711893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.712082 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.712162 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.713073 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.713194 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":51}
I20260927 15:17:05.714001 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.702462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.9.242.189:36101)
0927 15:17:05.703734 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.703738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.703748 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:17:05.703775 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.703778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.703815 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.703886 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.705411 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.706091 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.707181 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.708083 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.708171 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.709640 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.709644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.709647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.709703 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.711738 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.712156 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.712159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.712161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.712239 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.712932 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.712935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.712937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.713123 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.713246 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.713743 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.713854 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":75,"threads_started":1}
I20260927 15:17:05.712877 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.698174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.9.242.131:49955)
0927 15:17:05.698414 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.698427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.698646 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.699156 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.699163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.699759 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.700491 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.700512 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.701428 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.701431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.703245 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.703253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.703863 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.703867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.703973 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.704645 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.704659 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.706883 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.708784 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.708792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.708809 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.709128 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.709448 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.709451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.709453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.709624 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.710124 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.710130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.710456 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.711170 (+ 714us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.711513 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":510,"spinlock_wait_cycles":15104,"thread_start_us":82,"threads_started":1}
I20260927 15:17:05.715615 17535 heartbeater.cc:347] Connected to a master server at 127.9.242.190:35081
I20260927 15:17:05.715615 17536 heartbeater.cc:347] Connected to a master server at 127.9.242.189:36101
I20260927 15:17:05.715874 17535 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.715874 17536 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.716150 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.696525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:50551 (local address 127.9.242.188:36715)
0927 15:17:05.696658 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.696663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.697736 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 15:17:05.698894 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.698897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.698941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.698989 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.700722 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.701479 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.704267 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.704553 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.704655 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.707699 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.707703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.707706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.707730 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.709424 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.714692 (+ 5268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.714695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.714697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.714787 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.715284 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.715287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.715288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.715479 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.715558 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.715896 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.716005 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:17:05.716471 17536 heartbeater.cc:511] Master 127.9.242.189:36101 requested a full tablet report, sending...
I20260927 15:17:05.716516 17535 heartbeater.cc:511] Master 127.9.242.190:35081 requested a full tablet report, sending...
I20260927 15:17:05.717128 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.697039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.9.242.131:50551)
0927 15:17:05.697580 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.697624 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.698646 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.699153 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.699162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.699745 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:05.700619 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.700624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.701602 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.701605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.703108 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.703118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.704782 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.704786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.704873 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.705303 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:05.705325 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:05.707550 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:05.714343 (+ 6793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.714346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.714348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.714574 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.715089 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:05.715092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:05.715094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:05.715182 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:05.715625 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:05.715627 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:05.715717 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.716322 (+ 605us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.716460 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":132,"spinlock_wait_cycles":6016,"thread_start_us":654,"threads_started":1}
I20260927 15:17:05.717598 16297 ts_manager.cc:194] Re-registered known tserver with Master: 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:05.718235 17534 heartbeater.cc:347] Connected to a master server at 127.9.242.188:36715
I20260927 15:17:05.718326 17534 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:05.718328 16297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:59067
I20260927 15:17:05.718492 17534 heartbeater.cc:511] Master 127.9.242.188:36715 requested a full tablet report, sending...
I20260927 15:17:05.718489 16897 ts_manager.cc:194] Registered new tserver with Master: 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:05.719522 17081 ts_manager.cc:194] Registered new tserver with Master: 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:05.721555 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.702821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.9.242.188:36715)
0927 15:17:05.703335 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.703345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.704561 (+ 1216us) server_negotiation.cc:407] Connection header received
0927 15:17:05.705855 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.705858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.705909 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.706102 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.707566 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.708304 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.711292 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.711640 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.711731 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.715162 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.715165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.715168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.715190 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.716953 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.719475 (+ 2522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.719478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.719480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.719590 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.720553 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.720557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.720559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.720765 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.720884 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.721247 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.721372 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":79,"threads_started":1}
I20260927 15:17:05.722048 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.701828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.9.242.190:35081)
0927 15:17:05.703150 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.703154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.703394 (+ 240us) server_negotiation.cc:407] Connection header received
0927 15:17:05.703427 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.703429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.703483 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.703555 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.708416 (+ 4861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.709205 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.711830 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.712175 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.712261 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.714474 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.714479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.714482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.714506 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.718967 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.720130 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.720133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.720135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.720220 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.720590 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.720593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.720595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.720771 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.720884 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.721768 (+ 884us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.721888 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1250,"thread_start_us":79,"threads_started":1}
I20260927 15:17:05.722678 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.718203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.9.242.188:36715)
0927 15:17:05.718968 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.718972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.718985 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:05.719023 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.719026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.719081 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.719167 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:05.719441 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.720120 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.721409 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.721967 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.722061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.722214 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.722549 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.722604 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693}
I20260927 15:17:05.722681 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.717813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.0.0.1:43534)
0927 15:17:05.718020 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.718032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.718101 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.719197 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.719200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.719227 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:05.719316 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.719322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.720269 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.720272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.721300 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.721310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.722073 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.722076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.722362 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.722386 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.722494 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.722560 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260927 15:17:05.723474 17091 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 15:17:05.723852 17123 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f0a99a226304e7f83d3e945230fc57e. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.723965 17123 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: This master's current role is: LEARNER
I20260927 15:17:05.724802 16297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:05.728036 16297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:05.730403 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: NON_VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: true } } }
I20260927 15:17:05.730535 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:05.733368 17553 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: attempting to promote NON_VOTER fbe07224a450408e89e7487300022d12 to VOTER
I20260927 15:17:05.733731 17553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:05.736400 16907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Refusing update from remote peer 1f0a99a226304e7f83d3e945230fc57e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 15:17:05.736660 17090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 LEARNER]: Refusing update from remote peer 1f0a99a226304e7f83d3e945230fc57e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 15:17:05.737064 17413 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 15:17:05.737208 17553 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 15:17:05.738905 17413 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fbe07224a450408e89e7487300022d12 (127.9.242.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.739286 16907 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fbe07224a450408e89e7487300022d12 (127.9.242.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.739715 17091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fbe07224a450408e89e7487300022d12 (127.9.242.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.740238 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.740340 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:05.740500 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.740576 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:05.740480 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.740749 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:05.741879 17123 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.741990 17123 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:17:05.743613 17568 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f0a99a226304e7f83d3e945230fc57e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:05.743736 17568 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:17:05.748423 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.741677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:33009 (local address 127.0.0.1:59502)
0927 15:17:05.741785 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.741801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.741878 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.744522 (+ 2644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.744525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.744540 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:05.744624 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.744628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.745908 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.745911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.747116 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.747125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.748050 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.748054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.748143 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.748163 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.748218 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.748292 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":38}
I20260927 15:17:05.748469 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.742193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.9.242.130:33009)
0927 15:17:05.744137 (+ 1944us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.744147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.744167 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:17:05.744230 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.744234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.744408 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.744531 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:05.744733 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.745784 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.747260 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.747948 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.748041 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.748193 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.748255 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.748343 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1841,"thread_start_us":99,"threads_started":1}
I20260927 15:17:05.749356 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.741926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:33541 (local address 127.0.0.1:50592)
0927 15:17:05.742169 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.742181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.742271 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.745638 (+ 3367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.745640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.745677 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:05.745765 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.745772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.746773 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.746776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.747907 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.747916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.748927 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.748931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.749067 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.749091 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.749159 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.749227 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260927 15:17:05.749434 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.742554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.9.242.129:33541)
0927 15:17:05.744785 (+ 2231us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.744796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.744817 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:17:05.744884 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.744889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.745095 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.745705 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:05.745843 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.746688 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.748033 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.748811 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.748911 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.749042 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.749197 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.749287 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2131,"thread_start_us":86,"threads_started":1}
I20260927 15:17:05.750195 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.742198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:38333 (local address 127.0.0.1:36762)
0927 15:17:05.743192 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.743205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.743284 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.746285 (+ 3001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.746288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.746306 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:05.746388 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.746394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.747643 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.747646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.748826 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.748836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.749834 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.749837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.749930 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.749951 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.750009 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.750077 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":112,"threads_started":1}
I20260927 15:17:05.750295 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.742598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36762 (local address 127.9.242.131:38333)
0927 15:17:05.745880 (+ 3282us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.745891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.745910 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:17:05.745977 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.745981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.746153 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.746295 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:05.746488 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.747540 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.748942 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.749711 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.749833 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.750028 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.750073 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.750166 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3182,"thread_start_us":106,"threads_started":1}
I20260927 15:17:05.751791 17331 tablet_service.cc:1467] Processing CreateTablet for tablet c9213a4de7ff487caefb9a632334f6c1 (DEFAULT_TABLE table=test-table [id=03ab7a05bb5a4378a8a7f9bfbc36747a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:05.752194 17194 tablet_service.cc:1467] Processing CreateTablet for tablet c9213a4de7ff487caefb9a632334f6c1 (DEFAULT_TABLE table=test-table [id=03ab7a05bb5a4378a8a7f9bfbc36747a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:05.752746 17331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9213a4de7ff487caefb9a632334f6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:05.753053 17194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9213a4de7ff487caefb9a632334f6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:05.755661 17468 tablet_service.cc:1467] Processing CreateTablet for tablet c9213a4de7ff487caefb9a632334f6c1 (DEFAULT_TABLE table=test-table [id=03ab7a05bb5a4378a8a7f9bfbc36747a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:05.756659 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9213a4de7ff487caefb9a632334f6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:05.760965 17569 tablet_bootstrap.cc:492] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Bootstrap starting.
I20260927 15:17:05.761302 17570 tablet_bootstrap.cc:492] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Bootstrap starting.
I20260927 15:17:05.763163 17569 tablet_bootstrap.cc:654] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:05.763399 17570 tablet_bootstrap.cc:654] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:05.763991 17569 log.cc:824] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:05.764312 17570 log.cc:824] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:05.765184 17571 tablet_bootstrap.cc:492] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Bootstrap starting.
I20260927 15:17:05.766041 17569 tablet_bootstrap.cc:492] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: No bootstrap required, opened a new log
I20260927 15:17:05.766180 17570 tablet_bootstrap.cc:492] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: No bootstrap required, opened a new log
I20260927 15:17:05.766234 17569 ts_tablet_manager.cc:1397] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:05.766419 17570 ts_tablet_manager.cc:1397] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:05.767699 17571 tablet_bootstrap.cc:654] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:05.768466 17571 log.cc:824] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:05.770457 17571 tablet_bootstrap.cc:492] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: No bootstrap required, opened a new log
I20260927 15:17:05.770634 17571 ts_tablet_manager.cc:1397] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:17:05.770648 17569 raft_consensus.cc:348] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.770913 17569 raft_consensus.cc:374] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:05.770785 17570 raft_consensus.cc:348] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.771025 17570 raft_consensus.cc:374] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:05.770962 17569 raft_consensus.cc:729] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53258472abdd4851b8a46829b667eaab, State: Initialized, Role: FOLLOWER
I20260927 15:17:05.771083 17570 raft_consensus.cc:729] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8623336dd9ed46a79d3febf934f71a42, State: Initialized, Role: FOLLOWER
I20260927 15:17:05.771514 17569 consensus_queue.cc:260] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.771515 17570 consensus_queue.cc:260] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.772426 17569 ts_tablet_manager.cc:1428] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:17:05.772435 17570 ts_tablet_manager.cc:1428] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 15:17:05.775170 17398 heartbeater.cc:503] Master 127.9.242.190:35081 was elected leader, sending a full tablet report...
I20260927 15:17:05.775195 17261 heartbeater.cc:503] Master 127.9.242.190:35081 was elected leader, sending a full tablet report...
I20260927 15:17:05.774969 17571 raft_consensus.cc:348] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.776216 17571 raft_consensus.cc:374] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:05.776268 17571 raft_consensus.cc:729] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8765535ee06e4c7a937d61684587108d, State: Initialized, Role: FOLLOWER
I20260927 15:17:05.776682 17571 consensus_queue.cc:260] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.778488 17571 ts_tablet_manager.cc:1428] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 15:17:05.778681 17535 heartbeater.cc:503] Master 127.9.242.190:35081 was elected leader, sending a full tablet report...
W20260927 15:17:05.812466 17263 tablet.cc:2369] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:05.868841 17575 raft_consensus.cc:482] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:05.869077 17575 raft_consensus.cc:504] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.870335 17575 leader_election.cc:288] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53258472abdd4851b8a46829b667eaab (127.9.242.130:33009), 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:05.875129 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.871043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:54061 (local address 127.9.242.131:38333)
0927 15:17:05.871192 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.871196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.871293 (+ 97us) server_negotiation.cc:407] Connection header received
0927 15:17:05.871489 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.871492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.871542 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.871607 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:05.871797 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.872295 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.873970 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.874617 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.874727 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.874808 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.874942 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.875022 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:17:05.875140 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.870937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:38333 (local address 127.9.242.129:54061)
0927 15:17:05.871206 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.871220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.871330 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.871622 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.871624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.871639 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:05.871708 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.871713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.872395 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.872399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.873860 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.873866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.874719 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.874722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.874813 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.874838 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.874903 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.874980 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":86,"thread_start_us":79,"threads_started":1}
I20260927 15:17:05.875730 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9213a4de7ff487caefb9a632334f6c1" candidate_uuid: "8623336dd9ed46a79d3febf934f71a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8765535ee06e4c7a937d61684587108d" is_pre_election: true
I20260927 15:17:05.876014 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.870701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:53603 (local address 127.9.242.130:33009)
0927 15:17:05.871092 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.871097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.871124 (+ 27us) server_negotiation.cc:407] Connection header received
0927 15:17:05.871175 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.871177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.871224 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.871294 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:05.871589 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.872371 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.874862 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.875479 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.875603 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.875649 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.875826 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.875905 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260927 15:17:05.876051 17488 raft_consensus.cc:2454] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8623336dd9ed46a79d3febf934f71a42 in term 0.
I20260927 15:17:05.876362 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.870558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:33009 (local address 127.9.242.129:53603)
0927 15:17:05.870808 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:05.870824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:05.870924 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:05.871350 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:05.871353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:05.871373 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:05.871476 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.871482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.872466 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.872469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.873853 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:05.873866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.875594 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.875598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:05.875681 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.875703 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:17:05.876159 (+ 456us) client_negotiation.cc:240] Negotiation successful
0927 15:17:05.876237 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":51,"thread_start_us":87,"threads_started":1}
I20260927 15:17:05.876477 17149 leader_election.cc:302] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8623336dd9ed46a79d3febf934f71a42, 8765535ee06e4c7a937d61684587108d; no voters:
I20260927 15:17:05.876749 17575 raft_consensus.cc:2781] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:05.876830 17575 raft_consensus.cc:482] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:05.876874 17575 raft_consensus.cc:3037] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:05.877285 17351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9213a4de7ff487caefb9a632334f6c1" candidate_uuid: "8623336dd9ed46a79d3febf934f71a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53258472abdd4851b8a46829b667eaab" is_pre_election: true
I20260927 15:17:05.877534 17351 raft_consensus.cc:2454] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8623336dd9ed46a79d3febf934f71a42 in term 0.
I20260927 15:17:05.878127 17575 raft_consensus.cc:504] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.878429 17575 leader_election.cc:288] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [CANDIDATE]: Term 1 election: Requested vote from peers 53258472abdd4851b8a46829b667eaab (127.9.242.130:33009), 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:05.878788 17351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9213a4de7ff487caefb9a632334f6c1" candidate_uuid: "8623336dd9ed46a79d3febf934f71a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53258472abdd4851b8a46829b667eaab"
I20260927 15:17:05.878854 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9213a4de7ff487caefb9a632334f6c1" candidate_uuid: "8623336dd9ed46a79d3febf934f71a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8765535ee06e4c7a937d61684587108d"
I20260927 15:17:05.878894 17351 raft_consensus.cc:3037] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:05.878968 17488 raft_consensus.cc:3037] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:05.880010 17351 raft_consensus.cc:2454] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8623336dd9ed46a79d3febf934f71a42 in term 1.
I20260927 15:17:05.880239 17488 raft_consensus.cc:2454] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8623336dd9ed46a79d3febf934f71a42 in term 1.
I20260927 15:17:05.880496 17149 leader_election.cc:302] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53258472abdd4851b8a46829b667eaab, 8623336dd9ed46a79d3febf934f71a42; no voters:
I20260927 15:17:05.880676 17575 raft_consensus.cc:2781] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:05.880863 17575 raft_consensus.cc:686] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 1 LEADER]: Becoming Leader. State: Replica: 8623336dd9ed46a79d3febf934f71a42, State: Running, Role: LEADER
I20260927 15:17:05.881191 17575 consensus_queue.cc:237] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:05.884516 16297 catalog_manager.cc:5495] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129). New cstate: current_term: 1 leader_uuid: "8623336dd9ed46a79d3febf934f71a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } health_report { overall_health: UNKNOWN } } }
I20260927 15:17:05.896582 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.893519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.9.242.129:33541)
0927 15:17:05.893620 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.893624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.893636 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:17:05.893812 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.893814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.893868 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.893949 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:05.894123 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.894602 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.895367 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.895569 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.895640 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.895835 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:05.896254 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:05.896337 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.896409 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.896452 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:17:05.902165 17488 raft_consensus.cc:1261] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Refusing update from remote peer 8623336dd9ed46a79d3febf934f71a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:05.902242 17351 raft_consensus.cc:1261] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 1 FOLLOWER]: Refusing update from remote peer 8623336dd9ed46a79d3febf934f71a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:05.902796 17575 consensus_queue.cc:1035] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:05.903025 17580 consensus_queue.cc:1035] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:05.912091 17585 mvcc.cc:205] Tried to move back new op lower bound from 7333979037288976384 to 7333979037212188672. Current Snapshot: MvccSnapshot[applied={T|T < 7333979037288976384}]
I20260927 15:17:05.913208 17584 mvcc.cc:205] Tried to move back new op lower bound from 7333979037288976384 to 7333979037212188672. Current Snapshot: MvccSnapshot[applied={T|T < 7333979037288976384}]
I20260927 15:17:05.921034 16295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:05.921262 16295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:05.925200 16295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f0a99a226304e7f83d3e945230fc57e: Sending DeleteTablet for 3 replicas of tablet c9213a4de7ff487caefb9a632334f6c1
I20260927 15:17:05.926409 17194 tablet_service.cc:1514] Processing DeleteTablet for tablet c9213a4de7ff487caefb9a632334f6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:05 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:50592
I20260927 15:17:05.926470 17331 tablet_service.cc:1514] Processing DeleteTablet for tablet c9213a4de7ff487caefb9a632334f6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:05 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:59502
I20260927 15:17:05.926901 17593 tablet_replica.cc:330] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: stopping tablet replica
I20260927 15:17:05.926939 17594 tablet_replica.cc:330] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: stopping tablet replica
I20260927 15:17:05.927037 17468 tablet_service.cc:1514] Processing DeleteTablet for tablet c9213a4de7ff487caefb9a632334f6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:05 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:36762
I20260927 15:17:05.927302 17594 raft_consensus.cc:2229] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:05.927421 17593 raft_consensus.cc:2229] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:05.927621 17594 raft_consensus.cc:2258] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:05.927906 17593 raft_consensus.cc:2258] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:05.929279 17594 ts_tablet_manager.cc:1905] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:05.929512 17593 ts_tablet_manager.cc:1905] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:05.930724 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 16265
I20260927 15:17:05.931604 17595 tablet_replica.cc:330] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: stopping tablet replica
I20260927 15:17:05.931906 17595 raft_consensus.cc:2229] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:05.932201 17595 raft_consensus.cc:2258] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:05.932562 17594 ts_tablet_manager.cc:1918] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:05.932646 17594 log.cc:1196] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/c9213a4de7ff487caefb9a632334f6c1
I20260927 15:17:05.932948 17594 ts_tablet_manager.cc:1939] T c9213a4de7ff487caefb9a632334f6c1 P 53258472abdd4851b8a46829b667eaab: Deleting consensus metadata
I20260927 15:17:05.933585 17593 ts_tablet_manager.cc:1918] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:05.933684 17593 log.cc:1196] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/c9213a4de7ff487caefb9a632334f6c1
I20260927 15:17:05.933951 17593 ts_tablet_manager.cc:1939] T c9213a4de7ff487caefb9a632334f6c1 P 8623336dd9ed46a79d3febf934f71a42: Deleting consensus metadata
I20260927 15:17:05.936127 17595 ts_tablet_manager.cc:1905] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:17:05.937428 17287 connection.cc:535] server connection from 127.0.0.1:59502 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 15:17:05.937638 17148 connection.cc:535] server connection from 127.0.0.1:50592 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:17:05.938817 17595 ts_tablet_manager.cc:1918] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:05.938895 17595 log.cc:1196] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/c9213a4de7ff487caefb9a632334f6c1
I20260927 15:17:05.939179 17595 ts_tablet_manager.cc:1939] T c9213a4de7ff487caefb9a632334f6c1 P 8765535ee06e4c7a937d61684587108d: Deleting consensus metadata
W20260927 15:17:05.940222 17423 connection.cc:441] server connection from 127.0.0.1:36762 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36762 (request call id 1) could send its response
I20260927 15:17:05.947578 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.929351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.9.242.189:36101)
0927 15:17:05.929465 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.929470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.937924 (+ 8454us) server_negotiation.cc:407] Connection header received
0927 15:17:05.938035 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.938038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.938090 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.938156 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.939119 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.939935 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.941301 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.941697 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.941770 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.943640 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.943644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.943647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.943671 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.945813 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.946253 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.946257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.946259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.946349 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.946674 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.946677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.946679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.946882 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.946975 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.947289 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.947404 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 15:17:05.948976 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.939939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.9.242.188:36715)
0927 15:17:05.940044 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.940049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.940506 (+ 457us) server_negotiation.cc:407] Connection header received
0927 15:17:05.940679 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.940683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.940736 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.940788 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.942067 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.942809 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.943836 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.944208 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.944277 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.945705 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.945709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.945711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.945737 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.947296 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.947744 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.947747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.947749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.947826 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.948122 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.948126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.948128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.948312 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.948396 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.948719 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.948831 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:05.950578 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.941012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.9.242.189:36101)
0927 15:17:05.941122 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.941125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.941627 (+ 502us) server_negotiation.cc:407] Connection header received
0927 15:17:05.941801 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.941804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.941863 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.941913 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.942752 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.943486 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.944549 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.944863 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.944934 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.946538 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.946541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.946544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.946569 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.948018 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.948792 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.948795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.948797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.948868 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.949719 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.949722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.949724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.949914 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.950027 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.950339 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.950439 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:17:05.955711 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:05.929448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.9.242.188:36715)
0927 15:17:05.929585 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:05.929590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:05.948950 (+ 19360us) server_negotiation.cc:407] Connection header received
0927 15:17:05.948986 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:05.948989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:05.949029 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:05.949098 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:05.949754 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.950462 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.951314 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:05.951642 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:05.951746 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:05.952825 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:05.952829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:05.952832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:05.952853 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:05.954226 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.954565 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.954568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.954569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.954642 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:05.954848 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:05.954851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:05.954853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:05.955033 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:05.955148 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:05.955326 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 15:17:05.955556 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:17:06.915814 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:06.914936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:42863)
0927 15:17:06.915604 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:06.915700 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":75,"threads_started":1}
I20260927 15:17:06.916069 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:06.915583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:44635)
0927 15:17:06.915803 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:06.915890 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":62,"threads_started":1}
W20260927 15:17:06.916121 17535 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:35081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:06.916549 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:06.916190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:59055)
0927 15:17:06.916384 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:06.916461 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
W20260927 15:17:06.916596 17261 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:35081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
W20260927 15:17:06.916822 17398 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:35081 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:07.426338 17607 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f0a99a226304e7f83d3e945230fc57e)
I20260927 15:17:07.426532 17607 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:07.426779 17608 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1f0a99a226304e7f83d3e945230fc57e)
I20260927 15:17:07.427004 17608 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:07.427723 17607 leader_election.cc:288] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081), fb2f05ce55dd4176a0240f3673eb6268 (127.9.242.189:36101)
I20260927 15:17:07.427950 17608 leader_election.cc:288] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081), fbe07224a450408e89e7487300022d12 (127.9.242.188:36715)
I20260927 15:17:07.428396 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.427893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:34948)
0927 15:17:07.428212 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.428306 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":63,"threads_started":1}
W20260927 15:17:07.428570 17065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:07.429363 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.428067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:34964)
0927 15:17:07.428971 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.429273 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":49,"threads_started":1}
I20260927 15:17:07.429381 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.429183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:34970)
0927 15:17:07.429264 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.429314 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 15:17:07.429505 17065 leader_election.cc:334] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081): Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
W20260927 15:17:07.429508 16881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:07.430311 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.430093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:34974)
0927 15:17:07.430205 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.430253 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:17:07.430475 16881 leader_election.cc:334] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081): Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:07.432377 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.428251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36101 (local address 127.0.0.1:48504)
0927 15:17:07.428540 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.428554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:07.428664 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:07.429262 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:07.429264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:07.429283 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:07.429416 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:07.429422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.430627 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.430631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:07.431397 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:07.431405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.431993 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.431996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:07.432096 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:07.432115 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:17:07.432177 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:17:07.432239 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":4,"spinlock_wait_cycles":6912,"thread_start_us":43,"threads_started":1}
I20260927 15:17:07.432410 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.428495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.9.242.189:36101)
0927 15:17:07.429024 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:07.429028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:07.429042 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:17:07.429088 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:07.429091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:07.429135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:07.429709 (+ 574us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:07.429801 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.430496 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.431507 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.431890 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.431975 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:07.432076 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:07.432199 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 15:17:07.432273 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":467,"thread_start_us":67,"threads_started":1}
I20260927 15:17:07.432411 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.428162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36715 (local address 127.0.0.1:43564)
0927 15:17:07.429076 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.429088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:07.429189 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:07.429454 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:07.429456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:07.429472 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:07.429569 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:07.429575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.430573 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.430576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:07.431381 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:07.431390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.431996 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.432000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:07.432121 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:07.432139 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:17:07.432216 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:17:07.432272 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":22,"thread_start_us":59,"threads_started":1}
I20260927 15:17:07.432395 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.428607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.9.242.188:36715)
0927 15:17:07.428808 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:07.428812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:07.429164 (+ 352us) server_negotiation.cc:407] Connection header received
0927 15:17:07.429303 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:07.429306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:07.429348 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:07.429701 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:07.429794 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.430471 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.431492 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:07.431897 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:07.431986 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:07.432077 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:07.432236 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 15:17:07.432290 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":122,"thread_start_us":45,"threads_started":1}
I20260927 15:17:07.432813 16907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbe07224a450408e89e7487300022d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb2f05ce55dd4176a0240f3673eb6268" is_pre_election: true
I20260927 15:17:07.432926 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb2f05ce55dd4176a0240f3673eb6268" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbe07224a450408e89e7487300022d12" is_pre_election: true
I20260927 15:17:07.433027 16907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbe07224a450408e89e7487300022d12 in term 1.
I20260927 15:17:07.433070 17091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb2f05ce55dd4176a0240f3673eb6268 in term 1.
I20260927 15:17:07.433321 16882 leader_election.cc:302] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: fb2f05ce55dd4176a0240f3673eb6268, fbe07224a450408e89e7487300022d12; no voters: 1f0a99a226304e7f83d3e945230fc57e
I20260927 15:17:07.433420 17065 leader_election.cc:302] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: fb2f05ce55dd4176a0240f3673eb6268, fbe07224a450408e89e7487300022d12; no voters: 1f0a99a226304e7f83d3e945230fc57e
I20260927 15:17:07.433620 17607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:17:07.433635 17608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:17:07.433697 17607 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1f0a99a226304e7f83d3e945230fc57e)
I20260927 15:17:07.433706 17608 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1f0a99a226304e7f83d3e945230fc57e)
I20260927 15:17:07.433750 17607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:17:07.433751 17608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:17:07.434705 17607 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:07.434712 17608 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:07.435024 17608 leader_election.cc:288] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 2 election: Requested vote from peers 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081), fbe07224a450408e89e7487300022d12 (127.9.242.188:36715)
I20260927 15:17:07.435035 17607 leader_election.cc:288] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [CANDIDATE]: Term 2 election: Requested vote from peers 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081), fb2f05ce55dd4176a0240f3673eb6268 (127.9.242.189:36101)
I20260927 15:17:07.435333 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.435148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:34990)
0927 15:17:07.435232 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.435280 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 15:17:07.435446 16907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbe07224a450408e89e7487300022d12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb2f05ce55dd4176a0240f3673eb6268"
I20260927 15:17:07.435446 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb2f05ce55dd4176a0240f3673eb6268" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbe07224a450408e89e7487300022d12"
I20260927 15:17:07.435561 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.435147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:34976)
0927 15:17:07.435478 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.435516 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281}
I20260927 15:17:07.435592 16907 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fbe07224a450408e89e7487300022d12 in current term 2: Already voted for candidate fb2f05ce55dd4176a0240f3673eb6268 in this term.
I20260927 15:17:07.435591 17091 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fb2f05ce55dd4176a0240f3673eb6268 in current term 2: Already voted for candidate fbe07224a450408e89e7487300022d12 in this term.
I20260927 15:17:07.436141 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.435911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:34996)
0927 15:17:07.436036 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.436078 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 15:17:07.436259 16881 leader_election.cc:334] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081): Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:07.436347 16881 leader_election.cc:302] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb2f05ce55dd4176a0240f3673eb6268; no voters: 1f0a99a226304e7f83d3e945230fc57e, fbe07224a450408e89e7487300022d12
I20260927 15:17:07.436472 17608 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 15:17:07.436628 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.436338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:35004)
0927 15:17:07.436473 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.436572 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 15:17:07.436777 17065 leader_election.cc:334] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081): Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:07.436841 17065 leader_election.cc:302] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbe07224a450408e89e7487300022d12; no voters: 1f0a99a226304e7f83d3e945230fc57e, fb2f05ce55dd4176a0240f3673eb6268
I20260927 15:17:07.436952 17607 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 15:17:07.917373 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.916740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:50877)
0927 15:17:07.917191 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.917263 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":97,"threads_started":1}
I20260927 15:17:07.918256 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.917249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:56211)
0927 15:17:07.918106 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.918167 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":73,"threads_started":1}
I20260927 15:17:07.918426 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:07.917327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:49557)
0927 15:17:07.918210 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:07.918276 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":60,"threads_started":1}
I20260927 15:17:08.920094 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:08.919413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:57237)
0927 15:17:08.919909 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:08.919981 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":85,"threads_started":1}
I20260927 15:17:08.920320 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:08.918366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:57971)
0927 15:17:08.920128 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:08.920184 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1642,"thread_start_us":80,"threads_started":1}
I20260927 15:17:08.921000 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:08.919346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:60603)
0927 15:17:08.920764 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:08.920852 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1324,"thread_start_us":74,"threads_started":1}
I20260927 15:17:09.110664 17623 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:09.110800 17623 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:09.111243 17623 leader_election.cc:288] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081), fbe07224a450408e89e7487300022d12 (127.9.242.188:36715)
I20260927 15:17:09.111716 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb2f05ce55dd4176a0240f3673eb6268" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbe07224a450408e89e7487300022d12" is_pre_election: true
I20260927 15:17:09.111820 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.111428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:35242)
0927 15:17:09.111661 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.111730 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":73,"threads_started":1}
I20260927 15:17:09.111845 17091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb2f05ce55dd4176a0240f3673eb6268 in term 2.
I20260927 15:17:09.112120 16882 leader_election.cc:302] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fb2f05ce55dd4176a0240f3673eb6268, fbe07224a450408e89e7487300022d12; no voters:
I20260927 15:17:09.112291 17623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 15:17:09.112353 17623 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:09.112399 17623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:17:09.112576 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.112376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:35254)
0927 15:17:09.112473 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.112524 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 15:17:09.112669 16881 leader_election.cc:334] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081): Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:09.113430 17623 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:09.113755 17623 leader_election.cc:288] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 3 election: Requested vote from peers 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081), fbe07224a450408e89e7487300022d12 (127.9.242.188:36715)
I20260927 15:17:09.114015 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.113848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:35256)
0927 15:17:09.113924 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.113965 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 15:17:09.114050 17091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb2f05ce55dd4176a0240f3673eb6268" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbe07224a450408e89e7487300022d12"
I20260927 15:17:09.114171 17091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:17:09.114535 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.114366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:35270)
0927 15:17:09.114448 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.114489 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 15:17:09.114616 16881 leader_election.cc:334] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081): Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
I20260927 15:17:09.115141 17091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb2f05ce55dd4176a0240f3673eb6268 in term 3.
I20260927 15:17:09.115494 16882 leader_election.cc:302] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: fb2f05ce55dd4176a0240f3673eb6268, fbe07224a450408e89e7487300022d12; no voters: 1f0a99a226304e7f83d3e945230fc57e
I20260927 15:17:09.115648 17623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:17:09.115820 17623 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [term 3 LEADER]: Becoming Leader. State: Replica: fb2f05ce55dd4176a0240f3673eb6268, State: Running, Role: LEADER
I20260927 15:17:09.116056 17623 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } }
I20260927 15:17:09.117224 17626 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2f05ce55dd4176a0240f3673eb6268. Latest consensus state: current_term: 3 leader_uuid: "fb2f05ce55dd4176a0240f3673eb6268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:09.117317 17626 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:09.117555 17628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:09.119798 17628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:09.120167 17628 catalog_manager.cc:1259] Loaded cluster ID: 97ede2371e7c42ca9130d39d276086d8
I20260927 15:17:09.120219 17628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:09.120731 17628 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:09.121083 17628 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Loaded TSK: 0
I20260927 15:17:09.121770 17628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:17:09.165719 16897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:09.168721 16897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:09.171667 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.171440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:35298)
0927 15:17:09.171548 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.171599 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 15:17:09.171739 17091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [term 3 FOLLOWER]: Refusing update from remote peer fb2f05ce55dd4176a0240f3673eb6268: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 15:17:09.172173 17623 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 15:17:09.172326 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.172124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.0.0.1:35308)
0927 15:17:09.172223 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.172272 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:17:09.172420 16881 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 -> Peer 1f0a99a226304e7f83d3e945230fc57e (127.9.242.190:35081): Couldn't send request to peer 1f0a99a226304e7f83d3e945230fc57e. Status: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 15:17:09.173542 17630 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb2f05ce55dd4176a0240f3673eb6268. Latest consensus state: current_term: 3 leader_uuid: "fb2f05ce55dd4176a0240f3673eb6268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:09.173681 17630 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:17:09.174829 17626 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fb2f05ce55dd4176a0240f3673eb6268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:09.174938 17626 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:09.175274 17630 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fb2f05ce55dd4176a0240f3673eb6268" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f0a99a226304e7f83d3e945230fc57e" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35081 } } peers { permanent_uuid: "fb2f05ce55dd4176a0240f3673eb6268" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36101 } attrs { promote: false } } peers { permanent_uuid: "fbe07224a450408e89e7487300022d12" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36715 } attrs { promote: false } } }
I20260927 15:17:09.175401 17630 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbe07224a450408e89e7487300022d12 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:17:09.184967 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.180425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:33541 (local address 127.0.0.1:50622)
0927 15:17:09.180584 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.180598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:09.180682 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:09.181476 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:09.181479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:09.181498 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:09.181599 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.181605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.182828 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.182832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.183708 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.183717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.184617 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.184621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.184711 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.184731 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:17:09.184771 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 15:17:09.184825 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":50}
I20260927 15:17:09.185364 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.180884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:38333 (local address 127.0.0.1:36764)
0927 15:17:09.181092 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.181103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:09.181165 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:09.182147 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:09.182150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:09.182165 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:09.182250 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.182255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.182986 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.182989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.184143 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.184153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.185053 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.185057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.185139 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.185161 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:17:09.185203 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 15:17:09.185259 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260927 15:17:09.185554 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.180585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:33009 (local address 127.0.0.1:59512)
0927 15:17:09.180967 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.180977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:09.181054 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:09.181697 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:09.181700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:09.181715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:09.181796 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.181801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.182848 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.182851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.184149 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.184155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.184974 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.184978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.185067 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.185090 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:17:09.185334 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 15:17:09.185405 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":35,"thread_start_us":91,"threads_started":1}
I20260927 15:17:09.185606 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.181273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.9.242.130:33009)
0927 15:17:09.181481 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:09.181484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:09.181497 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:09.181541 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:09.181544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:09.181582 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:09.181670 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:09.181947 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.182723 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.184278 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.184882 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.184981 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.185047 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:09.185368 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 15:17:09.185486 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":55,"threads_started":1}
I20260927 15:17:09.185591 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.180556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.9.242.129:33541)
0927 15:17:09.181245 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:09.181257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:09.181270 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:09.181314 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:09.181316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:09.181354 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:09.181434 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:09.181922 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.182732 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.183860 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.184465 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.185247 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.185306 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:09.185340 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 15:17:09.185444 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":83,"threads_started":1}
I20260927 15:17:09.186003 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.180976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.9.242.131:38333)
0927 15:17:09.181965 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:09.181969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:09.181980 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:17:09.182019 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:09.182021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:09.182056 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:09.182128 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:09.182340 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.182890 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.184278 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.184943 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.185769 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.185826 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:09.185857 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 15:17:09.185911 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":55,"threads_started":1}
I20260927 15:17:09.186261 17194 tablet_service.cc:1467] Processing CreateTablet for tablet 2115bc8ec3184a16845fb630e8b5204c (DEFAULT_TABLE table=test-table [id=107764c2271f4002baa923ec10c0096f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:09.186586 17194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2115bc8ec3184a16845fb630e8b5204c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:09.186928 17331 tablet_service.cc:1467] Processing CreateTablet for tablet 2115bc8ec3184a16845fb630e8b5204c (DEFAULT_TABLE table=test-table [id=107764c2271f4002baa923ec10c0096f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:09.187171 17331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2115bc8ec3184a16845fb630e8b5204c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:09.189222 17640 tablet_bootstrap.cc:492] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42: Bootstrap starting.
I20260927 15:17:09.189284 17468 tablet_service.cc:1467] Processing CreateTablet for tablet 2115bc8ec3184a16845fb630e8b5204c (DEFAULT_TABLE table=test-table [id=107764c2271f4002baa923ec10c0096f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:09.189531 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2115bc8ec3184a16845fb630e8b5204c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:09.190055 17640 tablet_bootstrap.cc:654] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:09.190639 17641 tablet_bootstrap.cc:492] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Bootstrap starting.
I20260927 15:17:09.191095 17640 tablet_bootstrap.cc:492] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42: No bootstrap required, opened a new log
I20260927 15:17:09.191177 17640 ts_tablet_manager.cc:1397] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:09.191640 17643 tablet_bootstrap.cc:492] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Bootstrap starting.
I20260927 15:17:09.191592 17640 raft_consensus.cc:348] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.191700 17640 raft_consensus.cc:374] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:09.191730 17640 raft_consensus.cc:729] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8623336dd9ed46a79d3febf934f71a42, State: Initialized, Role: FOLLOWER
I20260927 15:17:09.191848 17640 consensus_queue.cc:260] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.192201 17640 ts_tablet_manager.cc:1428] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:09.192445 17643 tablet_bootstrap.cc:654] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:09.192727 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.192435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:37469)
0927 15:17:09.192595 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.192652 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 15:17:09.193574 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.193355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:46629)
0927 15:17:09.193464 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.193515 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 15:17:09.194079 17643 tablet_bootstrap.cc:492] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: No bootstrap required, opened a new log
I20260927 15:17:09.194166 17643 ts_tablet_manager.cc:1397] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:09.194346 17641 tablet_bootstrap.cc:654] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:09.194559 17643 raft_consensus.cc:348] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.194734 17643 raft_consensus.cc:374] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:09.194813 17643 raft_consensus.cc:729] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8765535ee06e4c7a937d61684587108d, State: Initialized, Role: FOLLOWER
I20260927 15:17:09.194896 17262 heartbeater.cc:503] Master 127.9.242.189:36101 was elected leader, sending a full tablet report...
I20260927 15:17:09.194957 17643 consensus_queue.cc:260] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.195294 17643 ts_tablet_manager.cc:1428] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:17:09.195920 17641 tablet_bootstrap.cc:492] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: No bootstrap required, opened a new log
I20260927 15:17:09.196000 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.195781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:34941)
0927 15:17:09.195887 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.195936 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:17:09.196024 17641 ts_tablet_manager.cc:1397] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 15:17:09.196415 17641 raft_consensus.cc:348] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.196519 17641 raft_consensus.cc:374] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:09.196550 17641 raft_consensus.cc:729] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53258472abdd4851b8a46829b667eaab, State: Initialized, Role: FOLLOWER
I20260927 15:17:09.196574 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.196340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:52145)
0927 15:17:09.196472 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.196520 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 15:17:09.196683 17641 consensus_queue.cc:260] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.197268 17641 ts_tablet_manager.cc:1428] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:09.197851 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.197561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:60825)
0927 15:17:09.197735 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.197788 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:17:09.198531 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.198312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:49827)
0927 15:17:09.198416 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.198475 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:17:09.198856 17537 tablet.cc:2369] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:09.199004 17536 heartbeater.cc:503] Master 127.9.242.189:36101 was elected leader, sending a full tablet report...
W20260927 15:17:09.267699 17400 tablet.cc:2369] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:09.596627 17648 raft_consensus.cc:482] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:09.596779 17648 raft_consensus.cc:504] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.597880 17648 leader_election.cc:288] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129:33541), 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:09.602608 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.598266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:40781 (local address 127.9.242.131:38333)
0927 15:17:09.598393 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:09.598397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:09.598782 (+ 385us) server_negotiation.cc:407] Connection header received
0927 15:17:09.598879 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:09.598881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:09.598927 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:09.599077 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:09.599198 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.599885 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.601379 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.602081 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.602183 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.602227 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:09.602424 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 15:17:09.602489 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:17:09.602608 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.598255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:50215 (local address 127.9.242.129:33541)
0927 15:17:09.598376 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:09.598381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:09.598399 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:17:09.598533 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:09.598537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:09.598587 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:09.598662 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:09.598970 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.599768 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.601379 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.602005 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.602137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.602191 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:09.602394 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 15:17:09.602471 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:17:09.602613 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.598148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:33541 (local address 127.9.242.130:50215)
0927 15:17:09.598292 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.598306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:09.598419 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:09.598716 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:09.598720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:09.598740 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:09.598871 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.598877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.599876 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.599878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.601230 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.601241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.602131 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.602135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.602254 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.602286 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 15:17:09.602366 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:17:09.602440 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":47,"spinlock_wait_cycles":1408}
I20260927 15:17:09.602631 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.598148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:38333 (local address 127.9.242.130:40781)
0927 15:17:09.598684 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.598697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:09.598798 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:09.599011 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:09.599013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:09.599028 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:09.599121 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.599125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.599969 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.599972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.601230 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:09.601241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.602234 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.602238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:09.602316 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.602333 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:17:09.602390 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:17:09.602442 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":214,"spinlock_wait_cycles":4224,"thread_start_us":140,"threads_started":1}
I20260927 15:17:09.603171 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2115bc8ec3184a16845fb630e8b5204c" candidate_uuid: "53258472abdd4851b8a46829b667eaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8765535ee06e4c7a937d61684587108d" is_pre_election: true
I20260927 15:17:09.603228 17214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2115bc8ec3184a16845fb630e8b5204c" candidate_uuid: "53258472abdd4851b8a46829b667eaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8623336dd9ed46a79d3febf934f71a42" is_pre_election: true
I20260927 15:17:09.603332 17488 raft_consensus.cc:2454] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53258472abdd4851b8a46829b667eaab in term 0.
I20260927 15:17:09.603439 17214 raft_consensus.cc:2454] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53258472abdd4851b8a46829b667eaab in term 0.
I20260927 15:17:09.603799 17286 leader_election.cc:302] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 53258472abdd4851b8a46829b667eaab, 8623336dd9ed46a79d3febf934f71a42, 8765535ee06e4c7a937d61684587108d; no voters:
I20260927 15:17:09.604127 17648 raft_consensus.cc:2781] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:09.604182 17648 raft_consensus.cc:482] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:09.604207 17648 raft_consensus.cc:3037] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:09.605168 17648 raft_consensus.cc:504] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.605484 17648 leader_election.cc:288] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [CANDIDATE]: Term 1 election: Requested vote from peers 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129:33541), 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333)
I20260927 15:17:09.605813 17214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2115bc8ec3184a16845fb630e8b5204c" candidate_uuid: "53258472abdd4851b8a46829b667eaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8623336dd9ed46a79d3febf934f71a42"
I20260927 15:17:09.605830 17488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2115bc8ec3184a16845fb630e8b5204c" candidate_uuid: "53258472abdd4851b8a46829b667eaab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8765535ee06e4c7a937d61684587108d"
I20260927 15:17:09.605922 17214 raft_consensus.cc:3037] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:09.605938 17488 raft_consensus.cc:3037] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:09.606767 17214 raft_consensus.cc:2454] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53258472abdd4851b8a46829b667eaab in term 1.
I20260927 15:17:09.606779 17488 raft_consensus.cc:2454] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53258472abdd4851b8a46829b667eaab in term 1.
I20260927 15:17:09.607029 17284 leader_election.cc:302] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53258472abdd4851b8a46829b667eaab, 8623336dd9ed46a79d3febf934f71a42; no voters:
I20260927 15:17:09.607236 17648 raft_consensus.cc:2781] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:09.607417 17648 raft_consensus.cc:686] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 1 LEADER]: Becoming Leader. State: Replica: 53258472abdd4851b8a46829b667eaab, State: Running, Role: LEADER
I20260927 15:17:09.607589 17399 heartbeater.cc:503] Master 127.9.242.189:36101 was elected leader, sending a full tablet report...
I20260927 15:17:09.607656 17648 consensus_queue.cc:237] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } }
I20260927 15:17:09.608184 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.607961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:54481)
0927 15:17:09.608057 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.608113 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 15:17:09.609963 16895 catalog_manager.cc:5495] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab reported cstate change: term changed from 0 to 1, leader changed from <none> to 53258472abdd4851b8a46829b667eaab (127.9.242.130). New cstate: current_term: 1 leader_uuid: "53258472abdd4851b8a46829b667eaab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53258472abdd4851b8a46829b667eaab" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 33009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 } health_report { overall_health: UNKNOWN } } }
I20260927 15:17:09.637071 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.633838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.9.242.130:33009)
0927 15:17:09.633967 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:09.633971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:09.633995 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:17:09.634074 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:09.634076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:09.634131 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:09.634225 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:09.634434 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.634929 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.635735 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:09.635948 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:09.636030 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:09.636280 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:09.636714 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:09.636814 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:09.636881 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 15:17:09.636933 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:17:09.639945 17488 raft_consensus.cc:1261] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Refusing update from remote peer 53258472abdd4851b8a46829b667eaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:09.640283 17214 raft_consensus.cc:1261] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 1 FOLLOWER]: Refusing update from remote peer 53258472abdd4851b8a46829b667eaab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:09.640448 17648 consensus_queue.cc:1035] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "8765535ee06e4c7a937d61684587108d" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:09.640786 17650 consensus_queue.cc:1035] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [LEADER]: Connected to new peer: Peer: permanent_uuid: "8623336dd9ed46a79d3febf934f71a42" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 33541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:09.642526 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.641991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:50867)
0927 15:17:09.642391 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.642451 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
I20260927 15:17:09.643136 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.642787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:51257)
0927 15:17:09.643009 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.643068 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260927 15:17:09.644702 17657 mvcc.cc:205] Tried to move back new op lower bound from 7333979052600455168 to 7333979052475580416. Current Snapshot: MvccSnapshot[applied={T|T < 7333979052600455168}]
I20260927 15:17:09.644838 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.644596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.129:43055)
0927 15:17:09.644730 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.644777 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 15:17:09.644894 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.644679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:52303)
0927 15:17:09.644788 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.644832 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:17:09.648406 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.648089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.130:57951)
0927 15:17:09.648287 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.648341 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 15:17:09.652974 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:09.649520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35081 (local address 127.9.242.131:34631)
0927 15:17:09.652842 (+ 3322us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.652900 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:35081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3247}
I20260927 15:17:09.653702 16897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:09.653861 16897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:09.657945 16897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb2f05ce55dd4176a0240f3673eb6268: Sending DeleteTablet for 3 replicas of tablet 2115bc8ec3184a16845fb630e8b5204c
I20260927 15:17:09.658560 17194 tablet_service.cc:1514] Processing DeleteTablet for tablet 2115bc8ec3184a16845fb630e8b5204c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:09 UTC) from {username='oryx', principal='oryx/127.9.242.189@KRBTEST.COM'} at 127.0.0.1:50622
I20260927 15:17:09.658902 17468 tablet_service.cc:1514] Processing DeleteTablet for tablet 2115bc8ec3184a16845fb630e8b5204c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:09 UTC) from {username='oryx', principal='oryx/127.9.242.189@KRBTEST.COM'} at 127.0.0.1:36764
I20260927 15:17:09.659215 17664 tablet_replica.cc:330] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42: stopping tablet replica
I20260927 15:17:09.659204 17331 tablet_service.cc:1514] Processing DeleteTablet for tablet 2115bc8ec3184a16845fb630e8b5204c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:09 UTC) from {username='oryx', principal='oryx/127.9.242.189@KRBTEST.COM'} at 127.0.0.1:59512
I20260927 15:17:09.659385 17664 raft_consensus.cc:2229] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:09.659515 17664 raft_consensus.cc:2258] T 2115bc8ec3184a16845fb630e8b5204c P 8623336dd9ed46a79d3febf934f71a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:09.659653 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 17042
I20260927 15:17:09.659731 17666 tablet_replica.cc:330] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: stopping tablet replica
I20260927 15:17:09.660367 17666 raft_consensus.cc:2229] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:09.660449 17665 tablet_replica.cc:330] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: stopping tablet replica
I20260927 15:17:09.660548 17665 raft_consensus.cc:2229] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:09.660655 17665 raft_consensus.cc:2258] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:09.660656 17666 raft_consensus.cc:2258] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:09.661262 17665 ts_tablet_manager.cc:1905] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:09.661262 17666 ts_tablet_manager.cc:1905] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:09.663643 17665 ts_tablet_manager.cc:1918] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:09.663643 17666 ts_tablet_manager.cc:1918] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:09.663712 17665 log.cc:1196] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/2115bc8ec3184a16845fb630e8b5204c
I20260927 15:17:09.663713 17666 log.cc:1196] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/2115bc8ec3184a16845fb630e8b5204c
I20260927 15:17:09.663957 17666 ts_tablet_manager.cc:1939] T 2115bc8ec3184a16845fb630e8b5204c P 53258472abdd4851b8a46829b667eaab: Deleting consensus metadata
I20260927 15:17:09.663957 17665 ts_tablet_manager.cc:1939] T 2115bc8ec3184a16845fb630e8b5204c P 8765535ee06e4c7a937d61684587108d: Deleting consensus metadata
I20260927 15:17:09.669346 16883 catalog_manager.cc:4841] TS 53258472abdd4851b8a46829b667eaab (127.9.242.130:33009): tablet 2115bc8ec3184a16845fb630e8b5204c (table test-table [id=107764c2271f4002baa923ec10c0096f]) successfully deleted
W20260927 15:17:09.669546 17636 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:09.668878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:33541 (local address 127.0.0.1:50630)
0927 15:17:09.669008 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:09.669024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:09.669116 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:09.669446 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:33541: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 15:17:09.669559 16883 catalog_manager.cc:4841] TS 8765535ee06e4c7a937d61684587108d (127.9.242.131:38333): tablet 2115bc8ec3184a16845fb630e8b5204c (table test-table [id=107764c2271f4002baa923ec10c0096f]) successfully deleted
W20260927 15:17:09.669754 16881 catalog_manager.cc:4584] TS 8623336dd9ed46a79d3febf934f71a42 (127.9.242.129:33541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2115bc8ec3184a16845fb630e8b5204c: Network error: Client connection negotiation failed: client connection to 127.9.242.129:33541: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:17:09.669956 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 17268
I20260927 15:17:09.675238 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 17404
I20260927 15:17:09.679981 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 16761
I20260927 15:17:09.684648 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 16861
2026-09-27T15:17:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7167 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:09 dist-test-slave-m4lt krb5kdc[17669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:09 dist-test-slave-m4lt krb5kdc[17669](info): set up 2 sockets
Sep 27 15:17:09 dist-test-slave-m4lt krb5kdc[17669](info): commencing operation
krb5kdc: starting...
W20260927 15:17:10.638046 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:17:10 dist-test-slave-m4lt krb5kdc[17669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab.
Entry for principal oryx/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab.
I20260927 15:17:10.754541 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43007
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab
--principal=oryx/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:43007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:10.875269 17682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:10.875542 17682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:10.875602 17682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:10.879544 17682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:10.879606 17682 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 15:17:10.879627 17682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:10.879650 17682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:10.879671 17682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:10.879695 17682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:10.885020 17682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:43007
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.190.keytab
--principal=oryx/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:10.886111 17682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:10.887216 17682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:17:10.895365 17682 server_base.cc:1034] running on GCE node
W20260927 15:17:10.895385 17688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:10.895393 17690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:10.895330 17687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:10.896245 17682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:10.897277 17682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:10.898429 17682 hybrid_clock.cc:648] HybridClock initialized: now 1790522230898404 us; error 37 us; skew 500 ppm
Sep 27 15:17:10 dist-test-slave-m4lt krb5kdc[17669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522230, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:10.900985 17682 init.cc:377] Logged in from keytab as oryx/127.9.242.190@KRBTEST.COM (short username oryx)
I20260927 15:17:10.901908 17682 webserver.cc:460] Webserver started at http://127.9.242.190:33813/ using document root <none> and password file <none>
I20260927 15:17:10.902421 17682 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:10.902475 17682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:10.902629 17682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:10.904578 17682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "32553d942d7946168f368d91f2ab8603"
format_stamp: "Formatted at 2026-09-27 15:17:10 on dist-test-slave-m4lt"
server_key: "d14de01314f099b7ab699b126ab72a7d"
server_key_iv: "e053dd8fa214e056b963ff10be7d647c"
server_key_version: "encryptionkey@0"
I20260927 15:17:10.905138 17682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "32553d942d7946168f368d91f2ab8603"
format_stamp: "Formatted at 2026-09-27 15:17:10 on dist-test-slave-m4lt"
server_key: "d14de01314f099b7ab699b126ab72a7d"
server_key_iv: "e053dd8fa214e056b963ff10be7d647c"
server_key_version: "encryptionkey@0"
I20260927 15:17:10.908725 17682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:17:10.911155 17697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:10.912396 17682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 15:17:10.912508 17682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "32553d942d7946168f368d91f2ab8603"
format_stamp: "Formatted at 2026-09-27 15:17:10 on dist-test-slave-m4lt"
server_key: "d14de01314f099b7ab699b126ab72a7d"
server_key_iv: "e053dd8fa214e056b963ff10be7d647c"
server_key_version: "encryptionkey@0"
I20260927 15:17:10.912606 17682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:10.937590 17682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:10.940466 17682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:10.940642 17682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:10.947907 17682 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:43007
I20260927 15:17:10.947918 17759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:43007 every 8 connection(s)
I20260927 15:17:10.948787 17682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:10.950625 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 17682
I20260927 15:17:10.950755 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:17:10.951040 10186 external_mini_cluster.cc:1428] Setting key fb67ca393edab39d8143b138409d0057
I20260927 15:17:10.951542 17760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:10.957172 17760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Bootstrap starting.
Sep 27 15:17:10 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:10.959357 17760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:10.960129 17760 log.cc:824] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:10.962148 17760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: No bootstrap required, opened a new log
I20260927 15:17:10.964885 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:10.952320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.9.242.190:43007)
0927 15:17:10.952729 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:10.952743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:10.952778 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:17:10.953426 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:10.953442 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:10.953812 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:10.954111 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:10.954848 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.955830 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.956631 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:10.957083 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:10.957222 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:10.959495 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:10.959500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:10.959520 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:10.959547 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:10.961929 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:10.962357 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:10.962362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:10.962368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:10.962513 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:10.962737 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:10.962740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:10.962741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:10.963233 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:10.963408 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:10.963644 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 15:17:10.963940 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":141,"threads_started":1}
I20260927 15:17:10.965839 17760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32553d942d7946168f368d91f2ab8603" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43007 } }
I20260927 15:17:10.966089 17760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:10.966144 17760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32553d942d7946168f368d91f2ab8603, State: Initialized, Role: FOLLOWER
I20260927 15:17:10.966581 17760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32553d942d7946168f368d91f2ab8603" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43007 } }
I20260927 15:17:10.966732 17760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:10.966790 17760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:10.966876 17760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:10.967777 17760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32553d942d7946168f368d91f2ab8603" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43007 } }
I20260927 15:17:10.968055 17760 leader_election.cc:302] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32553d942d7946168f368d91f2ab8603; no voters:
I20260927 15:17:10.968289 17760 leader_election.cc:288] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:10.968406 17765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:10.968593 17765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [term 1 LEADER]: Becoming Leader. State: Replica: 32553d942d7946168f368d91f2ab8603, State: Running, Role: LEADER
I20260927 15:17:10.968873 17765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32553d942d7946168f368d91f2ab8603" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43007 } }
I20260927 15:17:10.969349 17760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:10.970474 17766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32553d942d7946168f368d91f2ab8603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32553d942d7946168f368d91f2ab8603" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43007 } } }
I20260927 15:17:10.970614 17766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:10.970474 17767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32553d942d7946168f368d91f2ab8603. Latest consensus state: current_term: 1 leader_uuid: "32553d942d7946168f368d91f2ab8603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32553d942d7946168f368d91f2ab8603" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43007 } } }
I20260927 15:17:10.970709 17767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:10.971591 17774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:10.974761 17774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:10.980018 17774 catalog_manager.cc:1347] Generated new cluster ID: 93a1dabb9e294da9977265e3561f6bb8
I20260927 15:17:10.980083 17774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:10.994822 17774 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:10.996044 17774 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:11.016171 17774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Generated new TSK 0
I20260927 15:17:11.017023 17774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab.
Entry for principal oryx/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab.
I20260927 15:17:11.056346 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43007
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab
--principal=oryx/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:11.186739 17786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:11.186990 17786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:11.187033 17786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:11.191009 17786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:11.191072 17786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:11.191128 17786 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:17:11.191192 17786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:11.196411 17786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab
--principal=oryx/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:11.197408 17786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:11.198374 17786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:11.207316 17792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.207305 17794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.207307 17791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:11.207585 17786 server_base.cc:1034] running on GCE node
I20260927 15:17:11.208037 17786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:11.208633 17786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:11.209779 17786 hybrid_clock.cc:648] HybridClock initialized: now 1790522231209762 us; error 29 us; skew 500 ppm
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:11.212288 17786 init.cc:377] Logged in from keytab as oryx/127.9.242.129@KRBTEST.COM (short username oryx)
I20260927 15:17:11.213085 17786 webserver.cc:460] Webserver started at http://127.9.242.129:45887/ using document root <none> and password file <none>
I20260927 15:17:11.213589 17786 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:11.213647 17786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:11.213879 17786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:11.215924 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "31c407d027ec447887e51df5c844dce0"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "39c13936a144f60a9d02bccf8df546d9"
server_key_iv: "c0255e93578b7859fa2bc5f2ba60c915"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.216437 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "31c407d027ec447887e51df5c844dce0"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "39c13936a144f60a9d02bccf8df546d9"
server_key_iv: "c0255e93578b7859fa2bc5f2ba60c915"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.220111 17786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:17:11.222647 17801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.223943 17786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:11.224064 17786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "31c407d027ec447887e51df5c844dce0"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "39c13936a144f60a9d02bccf8df546d9"
server_key_iv: "c0255e93578b7859fa2bc5f2ba60c915"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.224187 17786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:11.245597 17786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:11.248348 17786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:11.248509 17786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:11.249917 17786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:11.249962 17786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.250002 17786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:11.250029 17786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:11.260118 17786 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:36141
I20260927 15:17:11.260200 17916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:36141 every 8 connection(s)
I20260927 15:17:11.260975 17786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:11.262737 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 17786
I20260927 15:17:11.262838 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:11.263137 10186 external_mini_cluster.cc:1428] Setting key 13eb131c8b6edc20b72896e5a7df6cf3
I20260927 15:17:11.263268 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.250672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:57415 (local address 127.9.242.190:43007)
0927 15:17:11.250818 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.250823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.251632 (+ 809us) server_negotiation.cc:407] Connection header received
0927 15:17:11.252642 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.252646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.252700 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.252772 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.254063 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.254663 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.255804 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.256034 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.256125 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.259535 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.259538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.259540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.259567 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.261112 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.261721 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.261725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.261727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.261815 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.262198 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.262201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.262203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.262385 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.262460 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.262981 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.263109 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:17:11.264072 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.251053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43007 (local address 127.9.242.129:57415)
0927 15:17:11.251481 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.251528 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.252413 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.252923 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.252932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.253412 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.253923 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.253937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.254774 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.254778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.255696 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.255705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.256169 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.256173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.256325 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.256902 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.256923 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.259358 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.261237 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.261244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.261260 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.261575 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.261923 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.261926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.261927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.262112 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.262470 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.262475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.262702 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.262873 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.263189 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260927 15:17:11.265506 17917 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43007
I20260927 15:17:11.265676 17917 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:11.266211 17917 heartbeater.cc:511] Master 127.9.242.190:43007 requested a full tablet report, sending...
I20260927 15:17:11.267602 17714 ts_manager.cc:194] Registered new tserver with Master: 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141)
I20260927 15:17:11.268900 17714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:57415
WARNING: no policy specified for oryx/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.130@KRBTEST.COM" created.
I20260927 15:17:11.274901 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.267591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:44737 (local address 127.9.242.190:43007)
0927 15:17:11.267700 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.267704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.267911 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:17:11.268013 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.268016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.268057 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.268134 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.268583 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.269406 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.270552 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.270937 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.271033 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.272022 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.272026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.272029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.272054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.273419 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.273814 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.273818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.273820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.273892 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.274124 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.274127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.274129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.274319 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.274400 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.274627 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.274745 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:17:11.274922 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.267537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43007 (local address 127.9.242.129:44737)
0927 15:17:11.267833 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.267848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.267936 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.268157 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.268160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.268347 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.268458 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.268466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.269511 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.269515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.270446 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.270456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.271066 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.271070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.271148 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.271581 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.271591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.271896 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.273509 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.273512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.273515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.273735 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.273970 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.273973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.273976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.274051 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.274400 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.274403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.274488 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.274572 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.274746 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab.
Entry for principal oryx/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab.
I20260927 15:17:11.288004 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43007
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab
--principal=oryx/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:11.394908 17926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:11.395177 17926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:11.395237 17926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:11.399180 17926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:11.399240 17926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:11.399293 17926 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:17:11.399346 17926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:11.404573 17926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab
--principal=oryx/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:11.405597 17926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:11.406587 17926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:11.415951 17932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.415967 17931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.415902 17934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:11.416137 17926 server_base.cc:1034] running on GCE node
I20260927 15:17:11.416715 17926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:11.417372 17926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:11.418519 17926 hybrid_clock.cc:648] HybridClock initialized: now 1790522231418482 us; error 54 us; skew 500 ppm
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:11.421082 17926 init.cc:377] Logged in from keytab as oryx/127.9.242.130@KRBTEST.COM (short username oryx)
I20260927 15:17:11.422010 17926 webserver.cc:460] Webserver started at http://127.9.242.130:35369/ using document root <none> and password file <none>
I20260927 15:17:11.422544 17926 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:11.422605 17926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:11.422798 17926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:11.424809 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "dd728d443eca4a9d9181b22af4060a97"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "7333900e6b1b0ef90679548f897368d7"
server_key_iv: "1d69959639b59d03bdba2f3c4b774241"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.425325 17926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "dd728d443eca4a9d9181b22af4060a97"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "7333900e6b1b0ef90679548f897368d7"
server_key_iv: "1d69959639b59d03bdba2f3c4b774241"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.429092 17926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:17:11.431435 17941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.432621 17926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:17:11.432740 17926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "dd728d443eca4a9d9181b22af4060a97"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "7333900e6b1b0ef90679548f897368d7"
server_key_iv: "1d69959639b59d03bdba2f3c4b774241"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.432837 17926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:11.444864 17926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:11.447592 17926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:11.447752 17926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:11.449085 17926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:11.449129 17926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.449168 17926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:11.449203 17926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:11.458818 17926 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:44389
I20260927 15:17:11.459061 18056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:44389 every 8 connection(s)
I20260927 15:17:11.459826 17926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:11.462081 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.449893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:45937 (local address 127.9.242.190:43007)
0927 15:17:11.449997 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.450000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.451013 (+ 1013us) server_negotiation.cc:407] Connection header received
0927 15:17:11.452006 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.452010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.452067 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.452125 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.453751 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.454553 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.455755 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.456095 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.456203 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.458460 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.458464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.458467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.458497 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.459982 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.460497 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.460501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.460503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.460639 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.461027 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.461030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.461031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.461221 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.461309 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.461866 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.461965 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:17:11.462944 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.450425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43007 (local address 127.9.242.130:45937)
0927 15:17:11.450878 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.450917 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.451774 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.452505 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.452515 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.452923 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.453584 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.453602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.454681 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.454684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.455652 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.455660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.456235 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.456238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.456367 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.456869 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.456891 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.458295 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.460088 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.460093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.460104 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.460382 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.460754 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.460756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.460758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.460914 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.461338 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.461344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.461600 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.461808 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.462137 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260927 15:17:11.463475 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 17926
I20260927 15:17:11.463581 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:11.463892 10186 external_mini_cluster.cc:1428] Setting key 5919ba24413124d32c537ea5a35942fd
I20260927 15:17:11.464205 18057 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43007
I20260927 15:17:11.464336 18057 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:11.464851 18057 heartbeater.cc:511] Master 127.9.242.190:43007 requested a full tablet report, sending...
I20260927 15:17:11.465799 17713 ts_manager.cc:194] Registered new tserver with Master: dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389)
I20260927 15:17:11.466454 17713 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:45937
I20260927 15:17:11.473599 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.466276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:36205 (local address 127.9.242.190:43007)
0927 15:17:11.466373 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.466376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.466935 (+ 559us) server_negotiation.cc:407] Connection header received
0927 15:17:11.467088 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.467091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.467131 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.467176 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.467735 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.468229 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.469347 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.469545 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.469689 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.470785 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.470787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.470789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.470809 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.472105 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.472535 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.472537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.472538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.472627 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.472869 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.472870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.472872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.473057 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.473126 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.473379 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.473462 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:17:11.473694 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.466225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43007 (local address 127.9.242.130:36205)
0927 15:17:11.466860 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.466876 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.466995 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.467299 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.467302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.467496 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.467610 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.467617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.468348 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.468352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.469183 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.469194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.469682 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.469686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.469772 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.470279 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.470292 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.470673 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.472207 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.472211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.472213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.472449 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.472700 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.472703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.472705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.472792 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.473140 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.473143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.473241 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.473347 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.473480 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab.
Entry for principal oryx/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab.
I20260927 15:17:11.490538 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43007
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab
--principal=oryx/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:11.596436 18066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:11.596709 18066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:11.596752 18066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:11.600721 18066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:11.600781 18066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:11.600818 18066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 15:17:11.600849 18066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:11.606066 18066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab
--principal=oryx/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.242.190:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:11.607023 18066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:11.607949 18066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:11.617236 18074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.617197 18071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:11.617202 18072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:11.617554 18066 server_base.cc:1034] running on GCE node
I20260927 15:17:11.617961 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:11.618558 18066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:11.619726 18066 hybrid_clock.cc:648] HybridClock initialized: now 1790522231619688 us; error 48 us; skew 500 ppm
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:11.622347 18066 init.cc:377] Logged in from keytab as oryx/127.9.242.131@KRBTEST.COM (short username oryx)
I20260927 15:17:11.623143 18066 webserver.cc:460] Webserver started at http://127.9.242.131:43915/ using document root <none> and password file <none>
I20260927 15:17:11.623651 18066 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:11.623713 18066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:11.623896 18066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:11.625861 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "2855eb91146447e9be8af267d05051f6"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "3f6148e12c8d9c57dc69d3b6ecda1f9e"
server_key_iv: "7cacbdc09b4d98f3150150c8c45a6ba5"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.626376 18066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "2855eb91146447e9be8af267d05051f6"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "3f6148e12c8d9c57dc69d3b6ecda1f9e"
server_key_iv: "7cacbdc09b4d98f3150150c8c45a6ba5"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.629930 18066 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:11.632200 18081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.633478 18066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:11.633577 18066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "2855eb91146447e9be8af267d05051f6"
format_stamp: "Formatted at 2026-09-27 15:17:11 on dist-test-slave-m4lt"
server_key: "3f6148e12c8d9c57dc69d3b6ecda1f9e"
server_key_iv: "7cacbdc09b4d98f3150150c8c45a6ba5"
server_key_version: "encryptionkey@0"
I20260927 15:17:11.633690 18066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:11.644672 18066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:11.647729 18066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:11.647902 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:11.649274 18066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:11.649322 18066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:11.649374 18066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:11.649405 18066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:11.659507 18066 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:36923
I20260927 15:17:11.659551 18196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:36923 every 8 connection(s)
I20260927 15:17:11.660362 18066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:11.662571 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.650198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:45383 (local address 127.9.242.190:43007)
0927 15:17:11.650307 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.650311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.651392 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 15:17:11.652518 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.652522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.652591 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.652689 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.653898 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.654434 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.656095 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.656443 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.656556 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.659356 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.659360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.659363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.659392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.660775 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.661317 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.661321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.661323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.661405 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.661680 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.661683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.661685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.661860 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.661936 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.662336 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.662432 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:17:11.663334 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.650764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43007 (local address 127.9.242.131:45383)
0927 15:17:11.651246 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.651286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.652299 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.652869 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.652876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.653206 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.653756 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.653771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.654556 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.654568 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.655956 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.655968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.656605 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.656609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.656801 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.657400 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.657420 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.659161 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.660897 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.660903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.660915 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.661224 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.661471 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.661472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.661474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.661601 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.661936 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.661940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.662149 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.662293 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.662541 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260927 15:17:11.664556 18197 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43007
I20260927 15:17:11.664739 18197 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:11.665337 18197 heartbeater.cc:511] Master 127.9.242.190:43007 requested a full tablet report, sending...
I20260927 15:17:11.666230 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 18066
I20260927 15:17:11.666353 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:11.666428 17714 ts_manager.cc:194] Registered new tserver with Master: 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:11.666656 10186 external_mini_cluster.cc:1428] Setting key 154b62cb06a7b67df643f99cc6f035b4
I20260927 15:17:11.667135 17714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:45383
I20260927 15:17:11.668499 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:17:11.673934 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.667412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:54993 (local address 127.9.242.190:43007)
0927 15:17:11.667501 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.667505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.667516 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:17:11.667542 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.667544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.667578 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.667647 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.668175 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.668902 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.669970 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.670219 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.670308 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.671361 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.671364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.671367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.671396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.672544 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.672895 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.672899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.672900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.672969 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.673193 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.673196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.673198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.673354 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.673431 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.673687 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.673785 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 15:17:11.673942 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.666574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43007 (local address 127.9.242.131:54993)
0927 15:17:11.666875 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.666890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.666969 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.667760 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.667762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.667949 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:11.668064 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.668071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.669024 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.669028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.669869 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.669879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.670322 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.670325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.670416 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.670886 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:11.670896 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:11.671247 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:11.672634 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.672637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.672639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.672817 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.673043 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:11.673046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:11.673048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:11.673121 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:11.673432 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:11.673436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:11.673519 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.673600 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.673754 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":133,"threads_started":1}
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:17:11.690603 17763 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:11.685027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.9.242.190:43007)
0927 15:17:11.685128 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.685131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.685142 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:17:11.685176 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.685179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.685213 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.685276 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.685950 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.686436 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.687219 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.687414 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.687478 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.690441 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35126: BlockingRecv error: recv got EOF from 127.0.0.1:35126 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 15:17:11 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:11.698743 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.691785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.9.242.190:43007)
0927 15:17:11.691874 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.691877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.692005 (+ 128us) server_negotiation.cc:407] Connection header received
0927 15:17:11.692065 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.692067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.692102 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.692161 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:11.692801 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.693317 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.694120 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.694340 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.694411 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.696329 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:11.696332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:11.696334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:11.696355 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:11.697534 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.697909 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.697911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.697913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.697977 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:11.698157 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:11.698159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:11.698160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:11.698320 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:11.698413 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.698546 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.698629 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 15:17:11.701197 17714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:11.704118 17714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:11.717237 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.711928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.9.242.130:44389)
0927 15:17:11.712528 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.712540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.712908 (+ 368us) server_negotiation.cc:407] Connection header received
0927 15:17:11.713101 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.713107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.713285 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.713421 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.713719 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.714528 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.715877 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.716614 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.716705 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.716936 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.716986 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.717071 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":90,"threads_started":1}
I20260927 15:17:11.717226 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.711827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44389 (local address 127.0.0.1:42246)
0927 15:17:11.712143 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.712188 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.712966 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.713421 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.713427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.713449 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.713570 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.713577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.714648 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.714651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.715773 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.715781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.716718 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.716721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.716824 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.716860 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.716957 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.717071 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":35,"thread_start_us":87,"threads_started":1}
I20260927 15:17:11.717810 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.712484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:36923 (local address 127.0.0.1:35814)
0927 15:17:11.712675 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.712685 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.712745 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.713785 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.713788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.713805 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.713888 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.713893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.715299 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.715302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.716433 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.716440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.717431 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.717434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.717515 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.717539 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.717595 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.717689 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260927 15:17:11.717885 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.712957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.9.242.131:36923)
0927 15:17:11.713374 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.713383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.713401 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:17:11.713460 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.713464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.713604 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.714024 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.714143 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.715182 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.716555 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.717331 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.717432 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.717575 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.717626 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.717743 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":104,"threads_started":1}
I20260927 15:17:11.718031 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.711908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36141 (local address 127.0.0.1:54240)
0927 15:17:11.712599 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.712612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.712743 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.713685 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.713688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.713705 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.713802 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.713808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.715125 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.715128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.716390 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.716400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.717648 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.717674 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.717770 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.717793 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.717856 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.717917 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":23,"thread_start_us":104,"threads_started":1}
I20260927 15:17:11.718601 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.712899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.9.242.129:36141)
0927 15:17:11.713224 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.713239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.713262 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:17:11.713331 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.713336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.713527 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.713720 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.713886 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.715005 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.716543 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.717333 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.717432 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.717576 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.718368 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.718467 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":93,"threads_started":1}
I20260927 15:17:11.720769 17851 tablet_service.cc:1467] Processing CreateTablet for tablet 5113293cff654a3b955cbaac0acbfba6 (DEFAULT_TABLE table=test-table [id=a1eaa7eb8347439f84cd4a9c2ba865b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.720749 17991 tablet_service.cc:1467] Processing CreateTablet for tablet 5113293cff654a3b955cbaac0acbfba6 (DEFAULT_TABLE table=test-table [id=a1eaa7eb8347439f84cd4a9c2ba865b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.720749 18131 tablet_service.cc:1467] Processing CreateTablet for tablet 5113293cff654a3b955cbaac0acbfba6 (DEFAULT_TABLE table=test-table [id=a1eaa7eb8347439f84cd4a9c2ba865b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.721740 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5113293cff654a3b955cbaac0acbfba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.721761 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5113293cff654a3b955cbaac0acbfba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.721755 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5113293cff654a3b955cbaac0acbfba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.728569 18225 tablet_bootstrap.cc:492] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Bootstrap starting.
I20260927 15:17:11.729791 18226 tablet_bootstrap.cc:492] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Bootstrap starting.
I20260927 15:17:11.729941 18227 tablet_bootstrap.cc:492] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Bootstrap starting.
I20260927 15:17:11.730895 18225 tablet_bootstrap.cc:654] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.731577 18227 tablet_bootstrap.cc:654] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.731738 18226 tablet_bootstrap.cc:654] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.731755 18225 log.cc:824] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:11.732306 18227 log.cc:824] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:11.732425 18226 log.cc:824] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:11.733906 18225 tablet_bootstrap.cc:492] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: No bootstrap required, opened a new log
I20260927 15:17:11.734146 18225 ts_tablet_manager.cc:1397] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:17:11.734243 18227 tablet_bootstrap.cc:492] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: No bootstrap required, opened a new log
I20260927 15:17:11.734375 18226 tablet_bootstrap.cc:492] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: No bootstrap required, opened a new log
I20260927 15:17:11.734411 18227 ts_tablet_manager.cc:1397] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:11.734536 18226 ts_tablet_manager.cc:1397] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:11.737624 18226 raft_consensus.cc:348] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.737910 18226 raft_consensus.cc:374] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.737979 18226 raft_consensus.cc:729] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.738034 18225 raft_consensus.cc:348] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.738312 18225 raft_consensus.cc:374] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.738373 18225 raft_consensus.cc:729] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c407d027ec447887e51df5c844dce0, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.738317 18227 raft_consensus.cc:348] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.738435 18226 consensus_queue.cc:260] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.738530 18227 raft_consensus.cc:374] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.738576 18227 raft_consensus.cc:729] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2855eb91146447e9be8af267d05051f6, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.738857 18225 consensus_queue.cc:260] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.738982 18227 consensus_queue.cc:260] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.739115 18057 heartbeater.cc:503] Master 127.9.242.190:43007 was elected leader, sending a full tablet report...
I20260927 15:17:11.739493 18226 ts_tablet_manager.cc:1428] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 15:17:11.739880 18225 ts_tablet_manager.cc:1428] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:17:11.739880 18227 ts_tablet_manager.cc:1428] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:11.740288 17917 heartbeater.cc:503] Master 127.9.242.190:43007 was elected leader, sending a full tablet report...
I20260927 15:17:11.740811 18197 heartbeater.cc:503] Master 127.9.242.190:43007 was elected leader, sending a full tablet report...
W20260927 15:17:11.762109 17918 tablet.cc:2369] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:11.771984 18232 raft_consensus.cc:482] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:11.772198 18232 raft_consensus.cc:504] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.773415 18232 leader_election.cc:288] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:11.778362 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.773750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:41485 (local address 127.9.242.130:44389)
0927 15:17:11.773868 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.773872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.774004 (+ 132us) server_negotiation.cc:407] Connection header received
0927 15:17:11.774262 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.774266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.774315 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.774382 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.774664 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.775440 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.777106 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.777726 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.777820 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.777867 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.778148 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.778241 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:17:11.778362 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.774157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:42387 (local address 127.9.242.131:36923)
0927 15:17:11.774273 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.774277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.774383 (+ 106us) server_negotiation.cc:407] Connection header received
0927 15:17:11.774478 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.774481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.774521 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.774574 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:11.774798 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.775529 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.777094 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.777704 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.777818 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.777867 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.778185 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.778247 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:17:11.778383 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.773711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44389 (local address 127.9.242.129:41485)
0927 15:17:11.773927 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.773941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.774042 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.774403 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.774406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.774419 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.774552 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.774565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.775543 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.775546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.776983 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.776994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.777813 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.777817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.777992 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.778055 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.778134 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.778210 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":123,"thread_start_us":62,"threads_started":1}
I20260927 15:17:11.778390 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.774058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:36923 (local address 127.9.242.129:42387)
0927 15:17:11.774303 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:11.774322 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:11.774403 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:11.774597 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:11.774600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:11.774616 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:11.774711 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.774715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.775625 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.775628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.776985 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:11.776994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.777813 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.777817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:11.777997 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.778054 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 15:17:11.778165 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:17:11.778213 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":141,"thread_start_us":102,"threads_started":1}
I20260927 15:17:11.779073 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5113293cff654a3b955cbaac0acbfba6" candidate_uuid: "31c407d027ec447887e51df5c844dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6" is_pre_election: true
I20260927 15:17:11.779085 18011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5113293cff654a3b955cbaac0acbfba6" candidate_uuid: "31c407d027ec447887e51df5c844dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd728d443eca4a9d9181b22af4060a97" is_pre_election: true
I20260927 15:17:11.779387 18011 raft_consensus.cc:2454] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c407d027ec447887e51df5c844dce0 in term 0.
I20260927 15:17:11.779388 18151 raft_consensus.cc:2454] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c407d027ec447887e51df5c844dce0 in term 0.
I20260927 15:17:11.779815 17802 leader_election.cc:302] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:11.780048 18232 raft_consensus.cc:2781] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:11.780129 18232 raft_consensus.cc:482] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:11.780169 18232 raft_consensus.cc:3037] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:11.781140 18232 raft_consensus.cc:504] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.781437 18232 leader_election.cc:288] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [CANDIDATE]: Term 1 election: Requested vote from peers dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:11.781802 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5113293cff654a3b955cbaac0acbfba6" candidate_uuid: "31c407d027ec447887e51df5c844dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6"
I20260927 15:17:11.781801 18011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5113293cff654a3b955cbaac0acbfba6" candidate_uuid: "31c407d027ec447887e51df5c844dce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd728d443eca4a9d9181b22af4060a97"
I20260927 15:17:11.781922 18151 raft_consensus.cc:3037] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:11.781922 18011 raft_consensus.cc:3037] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:11.783130 18151 raft_consensus.cc:2454] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c407d027ec447887e51df5c844dce0 in term 1.
I20260927 15:17:11.783421 17802 leader_election.cc:302] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2855eb91146447e9be8af267d05051f6, 31c407d027ec447887e51df5c844dce0; no voters:
I20260927 15:17:11.783474 18011 raft_consensus.cc:2454] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c407d027ec447887e51df5c844dce0 in term 1.
I20260927 15:17:11.783638 18232 raft_consensus.cc:2781] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:11.783818 18232 raft_consensus.cc:686] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 1 LEADER]: Becoming Leader. State: Replica: 31c407d027ec447887e51df5c844dce0, State: Running, Role: LEADER
I20260927 15:17:11.784152 18232 consensus_queue.cc:237] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.787279 17713 catalog_manager.cc:5495] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31c407d027ec447887e51df5c844dce0 (127.9.242.129). New cstate: current_term: 1 leader_uuid: "31c407d027ec447887e51df5c844dce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260927 15:17:11.815126 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:11.811912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.9.242.129:36141)
0927 15:17:11.812027 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:11.812031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:11.812044 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:11.812121 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:11.812123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:11.812173 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:11.812256 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:11.812427 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.812924 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.813756 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:11.814015 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:11.814145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:11.814341 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:11.814758 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:11.814837 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:11.814874 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 15:17:11.814923 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:17:11.820744 18151 raft_consensus.cc:1261] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Refusing update from remote peer 31c407d027ec447887e51df5c844dce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:11.820746 18011 raft_consensus.cc:1261] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Refusing update from remote peer 31c407d027ec447887e51df5c844dce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:11.821327 18232 consensus_queue.cc:1035] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:11.821522 18236 consensus_queue.cc:1035] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:11.828428 18239 mvcc.cc:205] Tried to move back new op lower bound from 7333979061531361280 to 7333979061390454784. Current Snapshot: MvccSnapshot[applied={T|T < 7333979061531361280}]
I20260927 15:17:11.839051 17713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:11.839200 17713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:11.841475 17713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Sending DeleteTablet for 3 replicas of tablet 5113293cff654a3b955cbaac0acbfba6
I20260927 15:17:11.842262 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet 5113293cff654a3b955cbaac0acbfba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:11 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54240
I20260927 15:17:11.842329 18131 tablet_service.cc:1514] Processing DeleteTablet for tablet 5113293cff654a3b955cbaac0acbfba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:11 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:35814
I20260927 15:17:11.842742 18250 tablet_replica.cc:330] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: stopping tablet replica
I20260927 15:17:11.842737 17991 tablet_service.cc:1514] Processing DeleteTablet for tablet 5113293cff654a3b955cbaac0acbfba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:11 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260927 15:17:11.843091 18250 raft_consensus.cc:2229] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:11.843147 18251 tablet_replica.cc:330] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: stopping tablet replica
I20260927 15:17:11.843502 18250 raft_consensus.cc:2258] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:11.843514 18251 raft_consensus.cc:2229] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:11.843971 18249 tablet_replica.cc:330] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: stopping tablet replica
I20260927 15:17:11.843983 18251 raft_consensus.cc:2258] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:11.844157 17713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 15:17:11.844276 18249 raft_consensus.cc:2229] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 15:17:11.844578 17713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:11.844607 18249 raft_consensus.cc:2258] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:11.846040 18251 ts_tablet_manager.cc:1905] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:11.846791 18250 ts_tablet_manager.cc:1905] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:11.848783 18251 ts_tablet_manager.cc:1918] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:11.848938 18251 log.cc:1196] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/5113293cff654a3b955cbaac0acbfba6
I20260927 15:17:11.849305 18250 ts_tablet_manager.cc:1918] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:11.849327 18251 ts_tablet_manager.cc:1939] T 5113293cff654a3b955cbaac0acbfba6 P dd728d443eca4a9d9181b22af4060a97: Deleting consensus metadata
I20260927 15:17:11.849399 18250 log.cc:1196] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/5113293cff654a3b955cbaac0acbfba6
I20260927 15:17:11.849854 18250 ts_tablet_manager.cc:1939] T 5113293cff654a3b955cbaac0acbfba6 P 31c407d027ec447887e51df5c844dce0: Deleting consensus metadata
I20260927 15:17:11.850057 18131 tablet_service.cc:1467] Processing CreateTablet for tablet 4025a2b7efd94b8ead3738b09ad3e106 (DEFAULT_TABLE table=test-table [id=e3b0581be9fa4436881c4f85ac7b5328]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.850344 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4025a2b7efd94b8ead3738b09ad3e106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.851015 17698 catalog_manager.cc:4841] TS dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389): tablet 5113293cff654a3b955cbaac0acbfba6 (table test-table [id=a1eaa7eb8347439f84cd4a9c2ba865b3]) successfully deleted
I20260927 15:17:11.852357 18249 ts_tablet_manager.cc:1905] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:11.852728 18227 tablet_bootstrap.cc:492] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Bootstrap starting.
I20260927 15:17:11.853227 17991 tablet_service.cc:1467] Processing CreateTablet for tablet 4025a2b7efd94b8ead3738b09ad3e106 (DEFAULT_TABLE table=test-table [id=e3b0581be9fa4436881c4f85ac7b5328]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.853472 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4025a2b7efd94b8ead3738b09ad3e106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.853645 18227 tablet_bootstrap.cc:654] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.853870 17699 catalog_manager.cc:4841] TS 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141): tablet 5113293cff654a3b955cbaac0acbfba6 (table test-table [id=a1eaa7eb8347439f84cd4a9c2ba865b3]) successfully deleted
I20260927 15:17:11.854374 17851 tablet_service.cc:1467] Processing CreateTablet for tablet 4025a2b7efd94b8ead3738b09ad3e106 (DEFAULT_TABLE table=test-table [id=e3b0581be9fa4436881c4f85ac7b5328]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:11.854636 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4025a2b7efd94b8ead3738b09ad3e106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:11.854846 18249 ts_tablet_manager.cc:1918] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:11.854924 18249 log.cc:1196] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/5113293cff654a3b955cbaac0acbfba6
I20260927 15:17:11.854984 18227 tablet_bootstrap.cc:492] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: No bootstrap required, opened a new log
I20260927 15:17:11.855054 18227 ts_tablet_manager.cc:1397] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:11.855270 18249 ts_tablet_manager.cc:1939] T 5113293cff654a3b955cbaac0acbfba6 P 2855eb91146447e9be8af267d05051f6: Deleting consensus metadata
I20260927 15:17:11.855468 18227 raft_consensus.cc:348] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.855579 18227 raft_consensus.cc:374] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.855610 18227 raft_consensus.cc:729] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2855eb91146447e9be8af267d05051f6, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.855724 18227 consensus_queue.cc:260] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.855975 18227 ts_tablet_manager.cc:1428] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:11.856447 17698 catalog_manager.cc:4841] TS 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923): tablet 5113293cff654a3b955cbaac0acbfba6 (table test-table [id=a1eaa7eb8347439f84cd4a9c2ba865b3]) successfully deleted
I20260927 15:17:11.856863 18225 tablet_bootstrap.cc:492] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: Bootstrap starting.
I20260927 15:17:11.857785 18225 tablet_bootstrap.cc:654] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.858930 18225 tablet_bootstrap.cc:492] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: No bootstrap required, opened a new log
I20260927 15:17:11.859021 18225 ts_tablet_manager.cc:1397] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:11.859368 18226 tablet_bootstrap.cc:492] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Bootstrap starting.
I20260927 15:17:11.859460 18225 raft_consensus.cc:348] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.859609 18225 raft_consensus.cc:374] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.859648 18225 raft_consensus.cc:729] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c407d027ec447887e51df5c844dce0, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.859767 18225 consensus_queue.cc:260] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.860004 18225 ts_tablet_manager.cc:1428] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:11.860319 18226 tablet_bootstrap.cc:654] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:11.861732 18226 tablet_bootstrap.cc:492] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: No bootstrap required, opened a new log
I20260927 15:17:11.861835 18226 ts_tablet_manager.cc:1397] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:11.862301 18226 raft_consensus.cc:348] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.862421 18226 raft_consensus.cc:374] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:11.862469 18226 raft_consensus.cc:729] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Initialized, Role: FOLLOWER
I20260927 15:17:11.862603 18226 consensus_queue.cc:260] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:11.862859 18226 ts_tablet_manager.cc:1428] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 15:17:11.911379 18198 tablet.cc:2369] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:17:11.961048 18058 tablet.cc:2369] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:17:12.066416 18231 raft_consensus.cc:482] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:12.066555 18231 raft_consensus.cc:504] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.067548 18231 leader_election.cc:288] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:12.072098 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.067961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:54509 (local address 127.9.242.129:36141)
0927 15:17:12.068182 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.068186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.068348 (+ 162us) server_negotiation.cc:407] Connection header received
0927 15:17:12.068497 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.068500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.068549 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.068625 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:12.068875 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.069601 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.070864 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.071387 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.071500 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.071568 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.071883 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.071957 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260927 15:17:12.072187 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.067931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:44009 (local address 127.9.242.131:36923)
0927 15:17:12.068098 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.068103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.068187 (+ 84us) server_negotiation.cc:407] Connection header received
0927 15:17:12.068384 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.068388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.068436 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.068511 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:12.068743 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.069537 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.071155 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.071707 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.071813 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.071857 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.072017 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.072084 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 15:17:12.072285 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.067795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:36923 (local address 127.9.242.130:44009)
0927 15:17:12.068103 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.068118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.068267 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.068518 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.068520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.068534 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:12.068656 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.068663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.069638 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.069641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.071050 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.071059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.071808 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.071812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.071909 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.071932 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.071997 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.072062 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":36,"thread_start_us":64,"threads_started":1}
I20260927 15:17:12.072451 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.067796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36141 (local address 127.9.242.130:54509)
0927 15:17:12.068281 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.068292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.068400 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.068639 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.068642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.068660 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:12.068753 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.068758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.069703 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.069706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.070755 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.070764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.071650 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.071653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.071750 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.071774 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.072239 (+ 465us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.072311 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":242,"thread_start_us":43,"threads_started":1}
I20260927 15:17:12.072935 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4025a2b7efd94b8ead3738b09ad3e106" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0" is_pre_election: true
I20260927 15:17:12.073082 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4025a2b7efd94b8ead3738b09ad3e106" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6" is_pre_election: true
I20260927 15:17:12.073200 18151 raft_consensus.cc:2454] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 0.
I20260927 15:17:12.073187 17871 raft_consensus.cc:2454] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 0.
I20260927 15:17:12.073597 17943 leader_election.cc:302] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2855eb91146447e9be8af267d05051f6, 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.073863 18231 raft_consensus.cc:2781] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:12.073928 18231 raft_consensus.cc:482] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:12.073966 18231 raft_consensus.cc:3037] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.075122 18231 raft_consensus.cc:504] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.075474 18231 leader_election.cc:288] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 election: Requested vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:12.075784 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4025a2b7efd94b8ead3738b09ad3e106" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0"
I20260927 15:17:12.075891 17871 raft_consensus.cc:3037] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.075942 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4025a2b7efd94b8ead3738b09ad3e106" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6"
I20260927 15:17:12.076037 18151 raft_consensus.cc:3037] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.077013 18151 raft_consensus.cc:2454] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 1.
I20260927 15:17:12.077013 17871 raft_consensus.cc:2454] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 1.
I20260927 15:17:12.078281 17943 leader_election.cc:302] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.078536 18231 raft_consensus.cc:2781] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:12.078732 18231 raft_consensus.cc:686] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 1 LEADER]: Becoming Leader. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Running, Role: LEADER
I20260927 15:17:12.078986 18231 consensus_queue.cc:237] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.081327 17713 catalog_manager.cc:5495] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd728d443eca4a9d9181b22af4060a97 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522230, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.190@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:12.119529 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.111591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43007 (local address 127.0.0.1:35142)
0927 15:17:12.111893 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.111909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.112055 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.112306 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.112309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.112754 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:12.112874 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.112882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.113913 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.113916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.114536 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.114544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.114938 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.114942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.115013 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.115498 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:12.115509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:12.116887 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:12.118286 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:12.118293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:12.118298 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:12.118503 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:12.118718 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:12.118720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:12.118721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:12.118788 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:12.119083 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:12.119087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:12.119185 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.119265 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.119374 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
I20260927 15:17:12.119541 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.111645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.9.242.190:43007)
0927 15:17:12.111878 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.111883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.111983 (+ 100us) server_negotiation.cc:407] Connection header received
0927 15:17:12.112141 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.112145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.112211 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.112291 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:12.113059 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.113810 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.114635 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.114850 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.114931 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.117003 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:12.117006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:12.117008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:12.117028 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:12.118193 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.118574 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.118576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.118578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.118643 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.118856 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.118858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.118859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.119013 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:12.119081 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.119304 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.119422 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":97}
I20260927 15:17:12.122841 17713 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:35142:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 15:17:12.123499 17713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:12.128312 17851 tablet_service.cc:1467] Processing CreateTablet for tablet f760f02e5f314cc1b397e987ce1b640b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2ab33aad49f14d6ab90f2812a5b06694]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:17:12.128312 17991 tablet_service.cc:1467] Processing CreateTablet for tablet f760f02e5f314cc1b397e987ce1b640b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2ab33aad49f14d6ab90f2812a5b06694]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:17:12.128454 18131 tablet_service.cc:1467] Processing CreateTablet for tablet f760f02e5f314cc1b397e987ce1b640b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2ab33aad49f14d6ab90f2812a5b06694]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 15:17:12.128692 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f760f02e5f314cc1b397e987ce1b640b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.128692 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f760f02e5f314cc1b397e987ce1b640b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.128859 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f760f02e5f314cc1b397e987ce1b640b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.131353 18226 tablet_bootstrap.cc:492] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: Bootstrap starting.
I20260927 15:17:12.131351 18225 tablet_bootstrap.cc:492] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: Bootstrap starting.
I20260927 15:17:12.131354 18227 tablet_bootstrap.cc:492] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: Bootstrap starting.
I20260927 15:17:12.132164 18226 tablet_bootstrap.cc:654] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.132160 18225 tablet_bootstrap.cc:654] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.132203 18227 tablet_bootstrap.cc:654] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.133241 18227 tablet_bootstrap.cc:492] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: No bootstrap required, opened a new log
I20260927 15:17:12.133327 18227 ts_tablet_manager.cc:1397] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:12.133356 18225 tablet_bootstrap.cc:492] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: No bootstrap required, opened a new log
I20260927 15:17:12.133432 18225 ts_tablet_manager.cc:1397] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:12.133790 18227 raft_consensus.cc:348] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.133898 18226 tablet_bootstrap.cc:492] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: No bootstrap required, opened a new log
I20260927 15:17:12.133904 18227 raft_consensus.cc:374] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.133935 18227 raft_consensus.cc:729] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2855eb91146447e9be8af267d05051f6, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.133965 18226 ts_tablet_manager.cc:1397] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 15:17:12.133842 18225 raft_consensus.cc:348] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.134069 18227 consensus_queue.cc:260] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.134163 18225 raft_consensus.cc:374] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.134213 18225 raft_consensus.cc:729] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c407d027ec447887e51df5c844dce0, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.134332 18227 ts_tablet_manager.cc:1428] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.134335 18225 consensus_queue.cc:260] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.134353 18226 raft_consensus.cc:348] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.134449 18226 raft_consensus.cc:374] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.134483 18226 raft_consensus.cc:729] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.134588 18225 ts_tablet_manager.cc:1428] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.134544 18233 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.134611 18226 consensus_queue.cc:260] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.134692 18233 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:17:12.134759 18246 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.134845 18226 ts_tablet_manager.cc:1428] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.134881 18246 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:17:12.135000 18231 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.135107 18231 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:17:12.274390 18231 raft_consensus.cc:482] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:12.274528 18231 raft_consensus.cc:504] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.274943 18231 leader_election.cc:288] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:12.275364 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f760f02e5f314cc1b397e987ce1b640b" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0" is_pre_election: true
I20260927 15:17:12.275429 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f760f02e5f314cc1b397e987ce1b640b" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6" is_pre_election: true
I20260927 15:17:12.275498 17871 raft_consensus.cc:2454] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 0.
I20260927 15:17:12.275575 18151 raft_consensus.cc:2454] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 0.
I20260927 15:17:12.275810 17943 leader_election.cc:302] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.275977 18231 raft_consensus.cc:2781] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:12.276033 18231 raft_consensus.cc:482] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:12.276077 18231 raft_consensus.cc:3037] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.276980 18231 raft_consensus.cc:504] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.277288 18231 leader_election.cc:288] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 election: Requested vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:12.277591 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f760f02e5f314cc1b397e987ce1b640b" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0"
I20260927 15:17:12.277637 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f760f02e5f314cc1b397e987ce1b640b" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6"
I20260927 15:17:12.277745 17871 raft_consensus.cc:3037] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.277779 18151 raft_consensus.cc:3037] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.278581 17871 raft_consensus.cc:2454] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 1.
I20260927 15:17:12.278724 18151 raft_consensus.cc:2454] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 1.
I20260927 15:17:12.278868 17943 leader_election.cc:302] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.279013 18231 raft_consensus.cc:2781] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:12.279102 18231 raft_consensus.cc:686] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [term 1 LEADER]: Becoming Leader. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Running, Role: LEADER
I20260927 15:17:12.279238 18231 consensus_queue.cc:237] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.279855 18257 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: TxnStatusTablet state changed. Reason: New leader dd728d443eca4a9d9181b22af4060a97. Latest consensus state: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.279939 18257 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:17:12.280169 18269 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 15:17:12.280303 18269 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 15:17:12.280882 17712 catalog_manager.cc:5495] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd728d443eca4a9d9181b22af4060a97 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260927 15:17:12.326459 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.321385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44389 (local address 127.0.0.1:42260)
0927 15:17:12.321601 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.321619 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.321772 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.322226 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.322228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.322235 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:17:12.322330 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.322334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.323895 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.323898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.324599 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.324608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.325202 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.325205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.325284 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.325331 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.326185 (+ 854us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.326200 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.326260 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.326344 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:17:12.326514 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.321563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.9.242.130:44389)
0927 15:17:12.321883 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.321890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.321906 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:17:12.321984 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.321987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.322062 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.322180 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:12.322437 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.323771 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.324713 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.325072 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.325169 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.325478 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.326075 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.326191 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.326273 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.326337 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":124,"threads_started":1}
I20260927 15:17:12.329546 18151 raft_consensus.cc:1261] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Refusing update from remote peer dd728d443eca4a9d9181b22af4060a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.329708 17871 raft_consensus.cc:1261] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Refusing update from remote peer dd728d443eca4a9d9181b22af4060a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.330092 18257 consensus_queue.cc:1035] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.330282 18231 consensus_queue.cc:1035] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.331822 18233 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: TxnStatusTablet state changed. Reason: New leader dd728d443eca4a9d9181b22af4060a97. Latest consensus state: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.331943 18233 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:17:12.331949 18246 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: TxnStatusTablet state changed. Reason: New leader dd728d443eca4a9d9181b22af4060a97. Latest consensus state: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.332065 18246 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:17:12.333874 18231 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.333981 18231 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:17:12.334775 18257 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.334872 18257 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P dd728d443eca4a9d9181b22af4060a97: This TxnStatusTablet replica's current role is: LEADER
I20260927 15:17:12.335716 18233 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.335731 18246 tablet_replica.cc:439] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } } }
I20260927 15:17:12.335816 18233 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P 2855eb91146447e9be8af267d05051f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:17:12.335826 18246 tablet_replica.cc:442] T f760f02e5f314cc1b397e987ce1b640b P 31c407d027ec447887e51df5c844dce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 15:17:12.339216 18273 mvcc.cc:205] Tried to move back new op lower bound from 7333979063616692224 to 7333979063417073664. Current Snapshot: MvccSnapshot[applied={T|T < 7333979063616692224}]
I20260927 15:17:12.342917 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.339522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42268 (local address 127.9.242.130:44389)
0927 15:17:12.339621 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.339625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.339863 (+ 238us) server_negotiation.cc:407] Connection header received
0927 15:17:12.339973 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.339976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.340023 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.340085 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:12.340311 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.340795 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.342014 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.342235 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.342309 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.342567 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.342629 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.342700 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.342771 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.342815 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:17:12.353968 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.350457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44389 (local address 127.9.242.130:45881)
0927 15:17:12.350724 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.350736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.350821 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.351069 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.351071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.351077 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:17:12.351175 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.351180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.352132 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.352134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.352743 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.352750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.353264 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.353267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.353337 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.353382 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.353667 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.353676 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.353746 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.353808 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260927 15:17:12.353976 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.350543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:45881 (local address 127.9.242.130:44389)
0927 15:17:12.350643 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.350648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.350793 (+ 145us) server_negotiation.cc:407] Connection header received
0927 15:17:12.350892 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.350895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.350939 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.351021 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:12.351256 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.352035 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.352833 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.353164 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.353253 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.353485 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.353559 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.353621 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.353765 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.353811 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260927 15:17:12.360073 17871 raft_consensus.cc:1261] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Refusing update from remote peer dd728d443eca4a9d9181b22af4060a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.360093 18151 raft_consensus.cc:1261] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Refusing update from remote peer dd728d443eca4a9d9181b22af4060a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.360522 18231 consensus_queue.cc:1035] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.360762 18231 consensus_queue.cc:1035] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.365023 18240 mvcc.cc:205] Tried to move back new op lower bound from 7333979063743188992 to 7333979062598254592. Current Snapshot: MvccSnapshot[applied={T|T < 7333979063743188992}]
I20260927 15:17:12.400964 17712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:12.401113 17712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:12.403261 17712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Sending DeleteTablet for 3 replicas of tablet 4025a2b7efd94b8ead3738b09ad3e106
I20260927 15:17:12.403810 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025a2b7efd94b8ead3738b09ad3e106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54240
I20260927 15:17:12.403884 17991 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025a2b7efd94b8ead3738b09ad3e106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260927 15:17:12.403929 18131 tablet_service.cc:1514] Processing DeleteTablet for tablet 4025a2b7efd94b8ead3738b09ad3e106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:35814
I20260927 15:17:12.404165 18289 tablet_replica.cc:330] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: stopping tablet replica
I20260927 15:17:12.404282 18289 raft_consensus.cc:2229] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:12.404402 18289 raft_consensus.cc:2258] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.404515 18290 tablet_replica.cc:330] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: stopping tablet replica
I20260927 15:17:12.404624 18290 raft_consensus.cc:2229] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:12.405122 18290 raft_consensus.cc:2258] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.405323 18289 ts_tablet_manager.cc:1905] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.405745 18290 ts_tablet_manager.cc:1905] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.408488 18289 ts_tablet_manager.cc:1918] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:17:12.408638 18289 log.cc:1196] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/4025a2b7efd94b8ead3738b09ad3e106
I20260927 15:17:12.408824 18290 ts_tablet_manager.cc:1918] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:17:12.408905 18290 log.cc:1196] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/4025a2b7efd94b8ead3738b09ad3e106
I20260927 15:17:12.409053 18289 ts_tablet_manager.cc:1939] T 4025a2b7efd94b8ead3738b09ad3e106 P 31c407d027ec447887e51df5c844dce0: Deleting consensus metadata
I20260927 15:17:12.409148 18290 ts_tablet_manager.cc:1939] T 4025a2b7efd94b8ead3738b09ad3e106 P dd728d443eca4a9d9181b22af4060a97: Deleting consensus metadata
I20260927 15:17:12.409804 17698 catalog_manager.cc:4841] TS dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389): tablet 4025a2b7efd94b8ead3738b09ad3e106 (table test-table [id=e3b0581be9fa4436881c4f85ac7b5328]) successfully deleted
I20260927 15:17:12.409904 18291 tablet_replica.cc:330] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: stopping tablet replica
I20260927 15:17:12.409825 17699 catalog_manager.cc:4841] TS 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141): tablet 4025a2b7efd94b8ead3738b09ad3e106 (table test-table [id=e3b0581be9fa4436881c4f85ac7b5328]) successfully deleted
I20260927 15:17:12.410041 18291 raft_consensus.cc:2229] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:12.410172 18291 raft_consensus.cc:2258] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.410811 18291 ts_tablet_manager.cc:1905] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.413610 18291 ts_tablet_manager.cc:1918] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:17:12.413717 18291 log.cc:1196] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/4025a2b7efd94b8ead3738b09ad3e106
I20260927 15:17:12.413956 18291 ts_tablet_manager.cc:1939] T 4025a2b7efd94b8ead3738b09ad3e106 P 2855eb91146447e9be8af267d05051f6: Deleting consensus metadata
I20260927 15:17:12.414674 17698 catalog_manager.cc:4841] TS 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923): tablet 4025a2b7efd94b8ead3738b09ad3e106 (table test-table [id=e3b0581be9fa4436881c4f85ac7b5328]) successfully deleted
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[17669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:17:12.434073 17763 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:12.428080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.9.242.190:43007)
0927 15:17:12.428195 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.428200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.428418 (+ 218us) server_negotiation.cc:407] Connection header received
0927 15:17:12.428572 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.428575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.428640 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.428719 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:12.429417 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.429973 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.430785 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.431005 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.431089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.433945 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35156: BlockingRecv error: recv got EOF from 127.0.0.1:35156 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[17669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM
I20260927 15:17:12.443671 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.435417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.9.242.190:43007)
0927 15:17:12.435510 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.435514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.435703 (+ 189us) server_negotiation.cc:407] Connection header received
0927 15:17:12.435809 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.435811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.435844 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.435905 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:12.436585 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.437206 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.438249 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.438449 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.438522 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.440825 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:12.440829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:12.440831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:12.440857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:12.442287 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.442638 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.442641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.442643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.442726 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:12.442930 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:12.442933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:12.442935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:12.443148 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:12.443271 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.443405 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.443512 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 15:17:12.446049 17712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:12.446441 17712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:12.450732 17851 tablet_service.cc:1467] Processing CreateTablet for tablet da9d2e31f1f24742964771538a0a8bd3 (DEFAULT_TABLE table=test-table [id=8925e1fb03874bea8301e1ef1f64a7d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:12.450863 17991 tablet_service.cc:1467] Processing CreateTablet for tablet da9d2e31f1f24742964771538a0a8bd3 (DEFAULT_TABLE table=test-table [id=8925e1fb03874bea8301e1ef1f64a7d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:12.451025 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9d2e31f1f24742964771538a0a8bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.450982 18131 tablet_service.cc:1467] Processing CreateTablet for tablet da9d2e31f1f24742964771538a0a8bd3 (DEFAULT_TABLE table=test-table [id=8925e1fb03874bea8301e1ef1f64a7d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:12.451120 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9d2e31f1f24742964771538a0a8bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.451213 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9d2e31f1f24742964771538a0a8bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.453339 18227 tablet_bootstrap.cc:492] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: Bootstrap starting.
I20260927 15:17:12.453336 18226 tablet_bootstrap.cc:492] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: Bootstrap starting.
I20260927 15:17:12.453339 18225 tablet_bootstrap.cc:492] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: Bootstrap starting.
I20260927 15:17:12.454177 18226 tablet_bootstrap.cc:654] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.454239 18227 tablet_bootstrap.cc:654] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.454378 18225 tablet_bootstrap.cc:654] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.455287 18226 tablet_bootstrap.cc:492] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: No bootstrap required, opened a new log
I20260927 15:17:12.455361 18226 ts_tablet_manager.cc:1397] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:12.455569 18225 tablet_bootstrap.cc:492] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: No bootstrap required, opened a new log
I20260927 15:17:12.455636 18225 ts_tablet_manager.cc:1397] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:12.455745 18227 tablet_bootstrap.cc:492] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: No bootstrap required, opened a new log
I20260927 15:17:12.455809 18227 ts_tablet_manager.cc:1397] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:12.455746 18226 raft_consensus.cc:348] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.455847 18226 raft_consensus.cc:374] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.455874 18226 raft_consensus.cc:729] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.455976 18226 consensus_queue.cc:260] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.455991 18225 raft_consensus.cc:348] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.456086 18225 raft_consensus.cc:374] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.456116 18225 raft_consensus.cc:729] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c407d027ec447887e51df5c844dce0, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.456213 18226 ts_tablet_manager.cc:1428] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.456192 18227 raft_consensus.cc:348] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.456277 18227 raft_consensus.cc:374] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.456238 18225 consensus_queue.cc:260] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.456305 18227 raft_consensus.cc:729] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2855eb91146447e9be8af267d05051f6, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.456404 18227 consensus_queue.cc:260] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.456476 18225 ts_tablet_manager.cc:1428] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.456625 18227 ts_tablet_manager.cc:1428] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.516129 18233 raft_consensus.cc:482] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:12.516285 18233 raft_consensus.cc:504] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.517542 18233 leader_election.cc:288] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389)
I20260927 15:17:12.520828 18231 raft_consensus.cc:482] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:12.520932 18231 raft_consensus.cc:504] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.521284 18231 leader_election.cc:288] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:12.521824 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0" is_pre_election: true
I20260927 15:17:12.521961 17871 raft_consensus.cc:2454] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 0.
I20260927 15:17:12.522485 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.518159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:43097 (local address 127.9.242.130:44389)
0927 15:17:12.518274 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.518278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.518360 (+ 82us) server_negotiation.cc:407] Connection header received
0927 15:17:12.518695 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.518698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.518749 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.518831 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:12.519149 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.519778 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.521181 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.521803 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.521886 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.521937 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.522282 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.522359 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:17:12.522512 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6" is_pre_election: true
I20260927 15:17:12.522630 18151 raft_consensus.cc:2454] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 0.
I20260927 15:17:12.522733 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.518014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:43507 (local address 127.9.242.129:36141)
0927 15:17:12.518134 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.518139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.518361 (+ 222us) server_negotiation.cc:407] Connection header received
0927 15:17:12.518568 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.518572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.518619 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.518697 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:12.519402 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.520178 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.521533 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.522142 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.522221 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.522266 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.522558 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.522631 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:17:12.522837 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.517835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44389 (local address 127.9.242.131:43097)
0927 15:17:12.518283 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.518298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.518450 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.518842 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.518846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.518864 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:12.518971 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.518977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.519876 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.519879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.521045 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.521054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.522047 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.522051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.522148 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.522171 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.522622 (+ 451us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.522697 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":22656,"thread_start_us":104,"threads_started":1}
I20260927 15:17:12.523267 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.517743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36141 (local address 127.9.242.131:43507)
0927 15:17:12.518283 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.518298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.518450 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.518746 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.518749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.518769 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:12.519281 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.519290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.520307 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.520310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.521414 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.521423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.522343 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.522346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.522431 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.522450 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.523069 (+ 619us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.523138 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"spinlock_wait_cycles":10240,"thread_start_us":78,"threads_started":1}
I20260927 15:17:12.523257 18011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd728d443eca4a9d9181b22af4060a97" is_pre_election: true
I20260927 15:17:12.523279 17943 leader_election.cc:302] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.523371 18011 raft_consensus.cc:2454] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2855eb91146447e9be8af267d05051f6 in term 0.
I20260927 15:17:12.523563 18231 raft_consensus.cc:2781] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:12.523610 18231 raft_consensus.cc:482] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:12.523645 18231 raft_consensus.cc:3037] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.523726 18082 leader_election.cc:302] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2855eb91146447e9be8af267d05051f6, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.523932 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0" is_pre_election: true
I20260927 15:17:12.524072 17871 raft_consensus.cc:2454] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2855eb91146447e9be8af267d05051f6 in term 0.
I20260927 15:17:12.524281 18233 raft_consensus.cc:2781] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:12.524333 18233 raft_consensus.cc:482] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:12.524367 18233 raft_consensus.cc:3037] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.524803 18231 raft_consensus.cc:504] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.525131 18231 leader_election.cc:288] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 election: Requested vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923)
I20260927 15:17:12.525413 18233 raft_consensus.cc:504] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.525492 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2855eb91146447e9be8af267d05051f6"
I20260927 15:17:12.525645 18151 raft_consensus.cc:2379] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dd728d443eca4a9d9181b22af4060a97 in current term 1: Already voted for candidate 2855eb91146447e9be8af267d05051f6 in this term.
I20260927 15:17:12.525756 18233 leader_election.cc:288] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 election: Requested vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389)
I20260927 15:17:12.525887 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "dd728d443eca4a9d9181b22af4060a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0"
I20260927 15:17:12.525979 17871 raft_consensus.cc:3037] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.526039 18011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd728d443eca4a9d9181b22af4060a97"
I20260927 15:17:12.526172 18011 raft_consensus.cc:2379] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2855eb91146447e9be8af267d05051f6 in current term 1: Already voted for candidate dd728d443eca4a9d9181b22af4060a97 in this term.
I20260927 15:17:12.526230 17870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9d2e31f1f24742964771538a0a8bd3" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0"
I20260927 15:17:12.526670 18083 leader_election.cc:302] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2855eb91146447e9be8af267d05051f6; no voters: 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97
I20260927 15:17:12.526885 18233 raft_consensus.cc:2726] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 15:17:12.527058 17871 raft_consensus.cc:2454] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd728d443eca4a9d9181b22af4060a97 in term 1.
I20260927 15:17:12.527305 17943 leader_election.cc:302] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31c407d027ec447887e51df5c844dce0, dd728d443eca4a9d9181b22af4060a97; no voters: 2855eb91146447e9be8af267d05051f6
I20260927 15:17:12.527453 18231 raft_consensus.cc:2781] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:12.527560 18231 raft_consensus.cc:686] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 1 LEADER]: Becoming Leader. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Running, Role: LEADER
I20260927 15:17:12.527689 18231 consensus_queue.cc:237] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.529202 17713 catalog_manager.cc:5495] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd728d443eca4a9d9181b22af4060a97 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "dd728d443eca4a9d9181b22af4060a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } health_report { overall_health: UNKNOWN } } }
I20260927 15:17:12.540287 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.537348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.9.242.130:44389)
0927 15:17:12.537456 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.537460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.537473 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:12.537625 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.537629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.537711 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.537773 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:12.538023 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.538521 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.539332 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.539568 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.539651 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.539899 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.539967 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.540022 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.540107 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.540151 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:17:12.541994 17871 raft_consensus.cc:1261] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Refusing update from remote peer dd728d443eca4a9d9181b22af4060a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.542026 18151 raft_consensus.cc:1261] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Refusing update from remote peer dd728d443eca4a9d9181b22af4060a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.542354 18278 consensus_queue.cc:1035] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.542493 18231 consensus_queue.cc:1035] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.549800 17711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:12.549953 17711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:12.552160 17711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Sending DeleteTablet for 3 replicas of tablet da9d2e31f1f24742964771538a0a8bd3
I20260927 15:17:12.552627 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet da9d2e31f1f24742964771538a0a8bd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54240
I20260927 15:17:12.552687 17850 tablet_service.cc:1514] Processing DeleteTablet for tablet da9d2e31f1f24742964771538a0a8bd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54240
I20260927 15:17:12.552805 18289 tablet_replica.cc:330] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: stopping tablet replica
I20260927 15:17:12.552911 18289 raft_consensus.cc:2229] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:12.552927 18131 tablet_service.cc:1514] Processing DeleteTablet for tablet da9d2e31f1f24742964771538a0a8bd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:35814
I20260927 15:17:12.553027 18289 raft_consensus.cc:2258] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.553069 18291 tablet_replica.cc:330] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: stopping tablet replica
I20260927 15:17:12.553054 17991 tablet_service.cc:1514] Processing DeleteTablet for tablet da9d2e31f1f24742964771538a0a8bd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260927 15:17:12.553155 18291 raft_consensus.cc:2229] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:12.553212 18290 tablet_replica.cc:330] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: stopping tablet replica
I20260927 15:17:12.553269 18291 raft_consensus.cc:2258] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.553315 18290 raft_consensus.cc:2229] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:12.553474 18290 raft_consensus.cc:2258] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.553726 18289 ts_tablet_manager.cc:1905] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.554152 18290 ts_tablet_manager.cc:1905] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.554375 18291 ts_tablet_manager.cc:1905] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.556023 18289 ts_tablet_manager.cc:1918] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:12.556085 18289 log.cc:1196] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/da9d2e31f1f24742964771538a0a8bd3
I20260927 15:17:12.556298 18289 ts_tablet_manager.cc:1939] T da9d2e31f1f24742964771538a0a8bd3 P 31c407d027ec447887e51df5c844dce0: Deleting consensus metadata
I20260927 15:17:12.556486 18290 ts_tablet_manager.cc:1918] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:12.556550 18290 log.cc:1196] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/da9d2e31f1f24742964771538a0a8bd3
I20260927 15:17:12.556620 18291 ts_tablet_manager.cc:1918] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:12.556684 18291 log.cc:1196] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/da9d2e31f1f24742964771538a0a8bd3
I20260927 15:17:12.556772 18290 ts_tablet_manager.cc:1939] T da9d2e31f1f24742964771538a0a8bd3 P dd728d443eca4a9d9181b22af4060a97: Deleting consensus metadata
I20260927 15:17:12.556926 18291 ts_tablet_manager.cc:1939] T da9d2e31f1f24742964771538a0a8bd3 P 2855eb91146447e9be8af267d05051f6: Deleting consensus metadata
I20260927 15:17:12.556913 17699 catalog_manager.cc:4841] TS 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141): tablet da9d2e31f1f24742964771538a0a8bd3 (table test-table [id=8925e1fb03874bea8301e1ef1f64a7d0]) successfully deleted
I20260927 15:17:12.557408 17698 catalog_manager.cc:4841] TS dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389): tablet da9d2e31f1f24742964771538a0a8bd3 (table test-table [id=8925e1fb03874bea8301e1ef1f64a7d0]) successfully deleted
W20260927 15:17:12.557566 17699 catalog_manager.cc:4808] TS 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141): delete failed for tablet da9d2e31f1f24742964771538a0a8bd3 because the tablet was not found. No further retry: Not found: Tablet not found: da9d2e31f1f24742964771538a0a8bd3
I20260927 15:17:12.557682 17698 catalog_manager.cc:4841] TS 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923): tablet da9d2e31f1f24742964771538a0a8bd3 (table test-table [id=8925e1fb03874bea8301e1ef1f64a7d0]) successfully deleted
I20260927 15:17:12.558698 17711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:12.559140 17711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:12.563380 17850 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13b91b0d7d4a709ae694ef28ee22c1 (DEFAULT_TABLE table=test-table [id=1f13166f52d24375b02cfd2d1b1f2dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:12.563643 17850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13b91b0d7d4a709ae694ef28ee22c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.563678 18131 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13b91b0d7d4a709ae694ef28ee22c1 (DEFAULT_TABLE table=test-table [id=1f13166f52d24375b02cfd2d1b1f2dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:12.563925 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13b91b0d7d4a709ae694ef28ee22c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.565716 18225 tablet_bootstrap.cc:492] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: Bootstrap starting.
I20260927 15:17:12.566294 17991 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13b91b0d7d4a709ae694ef28ee22c1 (DEFAULT_TABLE table=test-table [id=1f13166f52d24375b02cfd2d1b1f2dc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:12.566444 18225 tablet_bootstrap.cc:654] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.566529 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13b91b0d7d4a709ae694ef28ee22c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:12.566763 18227 tablet_bootstrap.cc:492] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: Bootstrap starting.
I20260927 15:17:12.567436 18225 tablet_bootstrap.cc:492] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: No bootstrap required, opened a new log
I20260927 15:17:12.567507 18225 ts_tablet_manager.cc:1397] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:12.567624 18227 tablet_bootstrap.cc:654] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.567884 18225 raft_consensus.cc:348] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.567981 18225 raft_consensus.cc:374] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.568007 18225 raft_consensus.cc:729] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c407d027ec447887e51df5c844dce0, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.568111 18225 consensus_queue.cc:260] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.568346 18225 ts_tablet_manager.cc:1428] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.568643 18226 tablet_bootstrap.cc:492] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: Bootstrap starting.
I20260927 15:17:12.569193 18227 tablet_bootstrap.cc:492] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: No bootstrap required, opened a new log
I20260927 15:17:12.569262 18227 ts_tablet_manager.cc:1397] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:12.569481 18226 tablet_bootstrap.cc:654] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:12.569736 18227 raft_consensus.cc:348] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.569839 18227 raft_consensus.cc:374] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.569870 18227 raft_consensus.cc:729] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2855eb91146447e9be8af267d05051f6, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.569980 18227 consensus_queue.cc:260] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.570322 18227 ts_tablet_manager.cc:1428] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.571141 18226 tablet_bootstrap.cc:492] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: No bootstrap required, opened a new log
I20260927 15:17:12.571215 18226 ts_tablet_manager.cc:1397] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 15:17:12.571612 18226 raft_consensus.cc:348] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.571717 18226 raft_consensus.cc:374] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:12.571750 18226 raft_consensus.cc:729] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd728d443eca4a9d9181b22af4060a97, State: Initialized, Role: FOLLOWER
I20260927 15:17:12.571861 18226 consensus_queue.cc:260] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.572093 18226 ts_tablet_manager.cc:1428] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 15:17:12.620736 18233 raft_consensus.cc:482] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:12.620860 18233 raft_consensus.cc:504] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.621263 18233 leader_election.cc:288] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389)
I20260927 15:17:12.621677 18011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b91b0d7d4a709ae694ef28ee22c1" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd728d443eca4a9d9181b22af4060a97" is_pre_election: true
I20260927 15:17:12.621680 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b91b0d7d4a709ae694ef28ee22c1" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0" is_pre_election: true
I20260927 15:17:12.621815 17871 raft_consensus.cc:2454] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2855eb91146447e9be8af267d05051f6 in term 0.
I20260927 15:17:12.621826 18011 raft_consensus.cc:2454] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2855eb91146447e9be8af267d05051f6 in term 0.
I20260927 15:17:12.622085 18082 leader_election.cc:302] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2855eb91146447e9be8af267d05051f6, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.622254 18233 raft_consensus.cc:2781] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:12.622308 18233 raft_consensus.cc:482] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:12.622349 18233 raft_consensus.cc:3037] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.623297 18233 raft_consensus.cc:504] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.623605 18233 leader_election.cc:288] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 election: Requested vote from peers 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141), dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389)
I20260927 15:17:12.623899 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b91b0d7d4a709ae694ef28ee22c1" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31c407d027ec447887e51df5c844dce0"
I20260927 15:17:12.624017 17871 raft_consensus.cc:3037] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.623974 18011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b91b0d7d4a709ae694ef28ee22c1" candidate_uuid: "2855eb91146447e9be8af267d05051f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd728d443eca4a9d9181b22af4060a97"
I20260927 15:17:12.624100 18011 raft_consensus.cc:3037] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:12.624946 17871 raft_consensus.cc:2454] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2855eb91146447e9be8af267d05051f6 in term 1.
I20260927 15:17:12.624991 18011 raft_consensus.cc:2454] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2855eb91146447e9be8af267d05051f6 in term 1.
I20260927 15:17:12.625248 18082 leader_election.cc:302] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2855eb91146447e9be8af267d05051f6, dd728d443eca4a9d9181b22af4060a97; no voters:
I20260927 15:17:12.625394 18233 raft_consensus.cc:2781] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:12.625571 18233 raft_consensus.cc:686] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2855eb91146447e9be8af267d05051f6, State: Running, Role: LEADER
I20260927 15:17:12.625819 18233 consensus_queue.cc:237] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } }
I20260927 15:17:12.628218 17711 catalog_manager.cc:5495] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2855eb91146447e9be8af267d05051f6 (127.9.242.131). New cstate: current_term: 1 leader_uuid: "2855eb91146447e9be8af267d05051f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2855eb91146447e9be8af267d05051f6" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 36923 } health_report { overall_health: HEALTHY } } }
I20260927 15:17:12.643152 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.639549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.9.242.131:36923)
0927 15:17:12.639796 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.639810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.639822 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:17:12.639887 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.639889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.639939 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.640039 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:12.640271 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.640822 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.641614 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.641898 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.641982 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.642313 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.642769 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.642872 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.642936 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.642990 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260927 15:17:12.652978 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.650237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:36633 (local address 127.9.242.130:44389)
0927 15:17:12.650340 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.650344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.650357 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:12.650415 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.650417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.650457 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.650515 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:12.650779 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.651276 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.652080 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.652298 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.652368 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.652658 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.652721 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.652766 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.652836 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.652873 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:17:12.653012 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.649951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:44389 (local address 127.9.242.131:36633)
0927 15:17:12.650256 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.650269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.650346 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.650557 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.650559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.650566 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:17:12.650700 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.650705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.651362 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.651364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.651988 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.651996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.652407 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.652410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.652485 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.652529 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.652777 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.652781 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.652823 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.652884 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260927 15:17:12.658229 18011 raft_consensus.cc:1261] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Refusing update from remote peer 2855eb91146447e9be8af267d05051f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.658303 17871 raft_consensus.cc:1261] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Refusing update from remote peer 2855eb91146447e9be8af267d05051f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:12.658603 18233 consensus_queue.cc:1035] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd728d443eca4a9d9181b22af4060a97" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.658784 18316 consensus_queue.cc:1035] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c407d027ec447887e51df5c844dce0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:12.677395 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.673885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:42661 (local address 127.9.242.131:36923)
0927 15:17:12.673991 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:12.673994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:12.674006 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:17:12.674049 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:12.674051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:12.674098 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:12.674160 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:12.674476 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.675192 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.676329 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.676630 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.676734 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.676943 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.677005 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.677057 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:12.677218 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 15:17:12.677266 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:17:12.677395 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:12.673728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:36923 (local address 127.9.242.130:42661)
0927 15:17:12.673842 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:12.673858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:12.673933 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:12.674264 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:12.674266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:12.674273 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 15:17:12.674365 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.674370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.675287 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.675291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.676208 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:12.676217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:12.676728 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:12.676732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:12.676812 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:12.676843 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 15:17:12.677130 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 15:17:12.677138 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 15:17:12.677195 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 15:17:12.677276 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 15:17:12.701731 17713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:12.701877 17713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:12.703994 17713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32553d942d7946168f368d91f2ab8603: Sending DeleteTablet for 3 replicas of tablet 5a13b91b0d7d4a709ae694ef28ee22c1
I20260927 15:17:12.704699 17850 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13b91b0d7d4a709ae694ef28ee22c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54240
I20260927 15:17:12.704759 18131 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13b91b0d7d4a709ae694ef28ee22c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:35814
I20260927 15:17:12.704890 18289 tablet_replica.cc:330] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: stopping tablet replica
I20260927 15:17:12.704886 17991 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13b91b0d7d4a709ae694ef28ee22c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:12 UTC) from {username='oryx', principal='oryx/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260927 15:17:12.704952 18291 tablet_replica.cc:330] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: stopping tablet replica
I20260927 15:17:12.705006 18289 raft_consensus.cc:2229] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:12.705053 18291 raft_consensus.cc:2229] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:12.705111 18289 raft_consensus.cc:2258] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.705127 18290 tablet_replica.cc:330] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: stopping tablet replica
I20260927 15:17:12.705224 18290 raft_consensus.cc:2229] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:12.705315 18290 raft_consensus.cc:2258] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.705308 18291 raft_consensus.cc:2258] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:12.705960 18289 ts_tablet_manager.cc:1905] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.706107 18291 ts_tablet_manager.cc:1905] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.706140 18290 ts_tablet_manager.cc:1905] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:12.708451 18289 ts_tablet_manager.cc:1918] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:17:12.708467 18291 ts_tablet_manager.cc:1918] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:17:12.708494 18290 ts_tablet_manager.cc:1918] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 15:17:12.708515 18289 log.cc:1196] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/5a13b91b0d7d4a709ae694ef28ee22c1
I20260927 15:17:12.708546 18290 log.cc:1196] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/5a13b91b0d7d4a709ae694ef28ee22c1
I20260927 15:17:12.708542 18291 log.cc:1196] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/5a13b91b0d7d4a709ae694ef28ee22c1
I20260927 15:17:12.708758 18289 ts_tablet_manager.cc:1939] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 31c407d027ec447887e51df5c844dce0: Deleting consensus metadata
I20260927 15:17:12.708777 18291 ts_tablet_manager.cc:1939] T 5a13b91b0d7d4a709ae694ef28ee22c1 P 2855eb91146447e9be8af267d05051f6: Deleting consensus metadata
I20260927 15:17:12.708777 18290 ts_tablet_manager.cc:1939] T 5a13b91b0d7d4a709ae694ef28ee22c1 P dd728d443eca4a9d9181b22af4060a97: Deleting consensus metadata
I20260927 15:17:12.709378 17699 catalog_manager.cc:4841] TS 31c407d027ec447887e51df5c844dce0 (127.9.242.129:36141): tablet 5a13b91b0d7d4a709ae694ef28ee22c1 (table test-table [id=1f13166f52d24375b02cfd2d1b1f2dc1]) successfully deleted
I20260927 15:17:12.709393 17698 catalog_manager.cc:4841] TS dd728d443eca4a9d9181b22af4060a97 (127.9.242.130:44389): tablet 5a13b91b0d7d4a709ae694ef28ee22c1 (table test-table [id=1f13166f52d24375b02cfd2d1b1f2dc1]) successfully deleted
I20260927 15:17:12.709622 17698 catalog_manager.cc:4841] TS 2855eb91146447e9be8af267d05051f6 (127.9.242.131:36923): tablet 5a13b91b0d7d4a709ae694ef28ee22c1 (table test-table [id=1f13166f52d24375b02cfd2d1b1f2dc1]) successfully deleted
I20260927 15:17:12.710109 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 17786
I20260927 15:17:12.716804 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 17926
I20260927 15:17:12.722882 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 18066
I20260927 15:17:12.727545 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 17682
2026-09-27T15:17:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3042 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[18329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[18329](info): set up 2 sockets
Sep 27 15:17:12 dist-test-slave-m4lt krb5kdc[18329](info): commencing operation
krb5kdc: starting...
W20260927 15:17:13.678076 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:17:13 dist-test-slave-m4lt krb5kdc[18329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:17:13.790457 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:44687
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:44687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:13.897598 18342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:13.897876 18342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:13.897914 18342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:13.901814 18342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:13.901866 18342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:13.901886 18342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:13.901903 18342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:13.901921 18342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:13.907058 18342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:44687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:13.908035 18342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:13.908974 18342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:13.916978 18347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:13.916978 18348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:13.916981 18350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:13.917186 18342 server_base.cc:1034] running on GCE node
I20260927 15:17:13.917945 18342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:13.919003 18342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:13.920177 18342 hybrid_clock.cc:648] HybridClock initialized: now 1790522233920153 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T15:17:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1200 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:13 dist-test-slave-m4lt krb5kdc[18355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:13 dist-test-slave-m4lt krb5kdc[18355](info): set up 2 sockets
Sep 27 15:17:13 dist-test-slave-m4lt krb5kdc[18355](info): commencing operation
krb5kdc: starting...
W20260927 15:17:14.886076 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:17:14 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:17:15.001497 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34307
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:15.112289 18368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:15.112573 18368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:15.112632 18368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:15.116721 18368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:15.116782 18368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:15.116809 18368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:15.116830 18368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:15.116856 18368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:15.122177 18368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:15.123261 18368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:15.124269 18368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:15.132477 18376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:15.132709 18368 server_base.cc:1034] running on GCE node
W20260927 15:17:15.132536 18374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:15.132570 18373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:15.133379 18368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:15.134555 18368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:15.135721 18368 hybrid_clock.cc:648] HybridClock initialized: now 1790522235135702 us; error 31 us; skew 500 ppm
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522235, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:15.144539 18368 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:17:15.145586 18368 webserver.cc:460] Webserver started at http://127.9.242.190:41785/ using document root <none> and password file <none>
I20260927 15:17:15.146190 18368 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:15.146260 18368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:15.146430 18368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:15.148626 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "a3672cf5d1444942abb8b1fd4bacc0e1"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "c594a525fa84cdbd454bb7e95e130637"
server_key_iv: "c6a450541bce7e605c8121cce24f4f45"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.149158 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "a3672cf5d1444942abb8b1fd4bacc0e1"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "c594a525fa84cdbd454bb7e95e130637"
server_key_iv: "c6a450541bce7e605c8121cce24f4f45"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.153124 18368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:17:15.155714 18383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.157024 18368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:15.157212 18368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "a3672cf5d1444942abb8b1fd4bacc0e1"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "c594a525fa84cdbd454bb7e95e130637"
server_key_iv: "c6a450541bce7e605c8121cce24f4f45"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.157330 18368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:15.185618 18368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:15.188845 18368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:15.189034 18368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:15.195900 18368 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:34307
I20260927 15:17:15.195919 18435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:34307 every 8 connection(s)
I20260927 15:17:15.196890 18368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:15.197614 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 18368
I20260927 15:17:15.197768 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:17:15.198074 10186 external_mini_cluster.cc:1428] Setting key efbe8f0fd0aee7976f619dc374392c1d
I20260927 15:17:15.200239 18436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:15.205684 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Bootstrap starting.
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:15.208065 18436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:15.208755 18436 log.cc:824] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:15.210873 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: No bootstrap required, opened a new log
I20260927 15:17:15.211776 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:15.199592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.9.242.190:34307)
0927 15:17:15.200003 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:15.200017 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:15.200060 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:17:15.200706 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:15.200724 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:15.201112 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:15.201428 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:15.202231 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.203213 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.204065 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.204530 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.204671 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:15.207259 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:15.207264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:15.207277 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:15.207304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:15.208833 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.209255 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.209260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.209264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.209366 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.209628 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.209630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.209632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.210163 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:15.210348 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:15.210559 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 15:17:15.210855 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260927 15:17:15.214421 18436 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:15.214632 18436 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:15.214675 18436 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3672cf5d1444942abb8b1fd4bacc0e1, State: Initialized, Role: FOLLOWER
I20260927 15:17:15.215020 18436 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:15.215122 18436 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:15.215162 18436 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:15.215236 18436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:15.216037 18436 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:15.216336 18436 leader_election.cc:302] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3672cf5d1444942abb8b1fd4bacc0e1; no voters:
I20260927 15:17:15.216585 18436 leader_election.cc:288] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:15.216753 18441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:15.216980 18441 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 LEADER]: Becoming Leader. State: Replica: a3672cf5d1444942abb8b1fd4bacc0e1, State: Running, Role: LEADER
I20260927 15:17:15.217278 18441 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:15.217799 18436 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:15.219233 18442 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } } }
I20260927 15:17:15.219358 18442 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:15.219728 18443 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3672cf5d1444942abb8b1fd4bacc0e1. Latest consensus state: current_term: 1 leader_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } } }
I20260927 15:17:15.219884 18443 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: This master's current role is: LEADER
W20260927 15:17:15.222584 18456 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:17:15.222651 18456 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:17:15.222723 18454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:15.223390 18454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:15.228801 18454 catalog_manager.cc:1347] Generated new cluster ID: 012b05f5a5dc469587caf925a1ee198a
I20260927 15:17:15.228869 18454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:15.243201 18454 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:15.244153 18454 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:15.255864 18454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Generated new TSK 0
I20260927 15:17:15.256628 18454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:17:15.288740 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34307
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:15.398087 18462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:15.398375 18462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:15.398437 18462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:15.402482 18462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:15.402558 18462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:15.402648 18462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:15.407938 18462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:15.409001 18462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:15.410058 18462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:15.419301 18470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:15.419301 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:15.419298 18468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:15.419718 18462 server_base.cc:1034] running on GCE node
I20260927 15:17:15.420189 18462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:15.420822 18462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:15.422047 18462 hybrid_clock.cc:648] HybridClock initialized: now 1790522235422026 us; error 33 us; skew 500 ppm
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522235, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:15.424746 18462 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:17:15.425704 18462 webserver.cc:460] Webserver started at http://127.9.242.129:46199/ using document root <none> and password file <none>
I20260927 15:17:15.426250 18462 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:15.426313 18462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:15.426492 18462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:15.428504 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "f626a1751e7f443e88673fb0f6ac76f8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "bbea2f92e10c73b9ec94a984f3a26797"
server_key_iv: "d61ba3a9b5fc1a7d92d740ad8adf56e6"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.429054 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "f626a1751e7f443e88673fb0f6ac76f8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "bbea2f92e10c73b9ec94a984f3a26797"
server_key_iv: "d61ba3a9b5fc1a7d92d740ad8adf56e6"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.432693 18462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:17:15.435171 18477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.436359 18462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:15.436460 18462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "f626a1751e7f443e88673fb0f6ac76f8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "bbea2f92e10c73b9ec94a984f3a26797"
server_key_iv: "d61ba3a9b5fc1a7d92d740ad8adf56e6"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.436547 18462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:15.451568 18462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:15.454463 18462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:15.454625 18462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:15.455156 18462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:15.455964 18462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:15.456019 18462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.456063 18462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:15.456089 18462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.464344 18462 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:39797
I20260927 15:17:15.464368 18590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:39797 every 8 connection(s)
I20260927 15:17:15.465220 18462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522235, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:15.475049 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 18462
I20260927 15:17:15.475170 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:15.475472 10186 external_mini_cluster.cc:1428] Setting key 91c005b8cb265993c6be83aed9884dbd
I20260927 15:17:15.478134 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:15.467124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:51763 (local address 127.9.242.190:34307)
0927 15:17:15.467273 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:15.467276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:15.468228 (+ 952us) server_negotiation.cc:407] Connection header received
0927 15:17:15.469074 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:15.469078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:15.469128 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:15.469204 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:15.470446 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.470988 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.472119 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.472350 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.472427 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:15.474772 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:15.474775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:15.474777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:15.474800 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:15.476216 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.476675 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.476678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.476680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.476763 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.477099 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.477101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.477102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.477267 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:15.477335 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:15.477927 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 15:17:15.478022 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:17:15.479103 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:15.467420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:51763)
0927 15:17:15.468070 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:15.468114 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:15.468889 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:15.469360 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:15.469369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:15.469819 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:15.470316 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:15.470329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.471115 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.471118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:15.472004 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:15.472013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.472466 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.472468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:15.472603 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:15.473162 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:15.473180 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:15.474616 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:15.476318 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:15.476323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:15.476334 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:15.476581 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:15.476847 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:15.476850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:15.476852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:15.477019 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:15.477363 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:15.477369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:15.477701 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 15:17:15.477900 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 15:17:15.478214 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":113,"threads_started":1}
I20260927 15:17:15.480304 18591 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34307
I20260927 15:17:15.480530 18591 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:15.481010 18591 heartbeater.cc:511] Master 127.9.242.190:34307 requested a full tablet report, sending...
I20260927 15:17:15.482458 18400 ts_manager.cc:194] Registered new tserver with Master: f626a1751e7f443e88673fb0f6ac76f8 (127.9.242.129:39797)
I20260927 15:17:15.483582 18400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:51763
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:17:15.500213 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34307
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:15.605855 18596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:15.606112 18596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:15.606169 18596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:15.610083 18596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:15.610136 18596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:15.610210 18596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:15.615320 18596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:15.616322 18596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:15.617280 18596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:15.626538 18604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:15.626494 18601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:15.626500 18602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:15.626804 18596 server_base.cc:1034] running on GCE node
I20260927 15:17:15.627259 18596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:15.627909 18596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:15.629074 18596 hybrid_clock.cc:648] HybridClock initialized: now 1790522235629045 us; error 47 us; skew 500 ppm
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522235, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:15.631925 18596 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:17:15.632851 18596 webserver.cc:460] Webserver started at http://127.9.242.130:34511/ using document root <none> and password file <none>
I20260927 15:17:15.633386 18596 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:15.633452 18596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:15.633713 18596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:15.635811 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "40e1d55a27b6460fa6cad54565c0f4a8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "2b4cd1d496f611e3cd339a38497c0aa9"
server_key_iv: "f717bbad8f5b7c5557b9443af59a67d8"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.636348 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "40e1d55a27b6460fa6cad54565c0f4a8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "2b4cd1d496f611e3cd339a38497c0aa9"
server_key_iv: "f717bbad8f5b7c5557b9443af59a67d8"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.640270 18596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:17:15.642735 18611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.644018 18596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:15.644137 18596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "40e1d55a27b6460fa6cad54565c0f4a8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "2b4cd1d496f611e3cd339a38497c0aa9"
server_key_iv: "f717bbad8f5b7c5557b9443af59a67d8"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.644238 18596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:15.661962 18596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:15.664705 18596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:15.664877 18596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:15.665431 18596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:15.666249 18596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:15.666294 18596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.666345 18596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:15.666369 18596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.674437 18596 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:42413
I20260927 15:17:15.674470 18724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:42413 every 8 connection(s)
I20260927 15:17:15.675284 18596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:15.676003 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 18596
I20260927 15:17:15.676118 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:15.676375 10186 external_mini_cluster.cc:1428] Setting key 0166fbfebcdc3bc9e719b01263562083
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522235, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:17:15.689388 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:15.677041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:41637 (local address 127.9.242.190:34307)
0927 15:17:15.677161 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:15.677164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:15.678519 (+ 1355us) server_negotiation.cc:407] Connection header received
0927 15:17:15.679693 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:15.679696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:15.679751 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:15.679833 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:15.681126 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.681951 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.683173 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.683512 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.683623 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:15.686160 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:15.686172 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:15.686175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:15.686204 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:15.687429 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.687980 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.687984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.687986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.688059 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.688352 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.688355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.688357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.688534 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:15.688615 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:15.689099 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 15:17:15.689224 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:17:15.690258 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:15.677537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:41637)
0927 15:17:15.678361 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:15.678416 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:15.679438 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:15.679991 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:15.679997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:15.680358 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:15.680963 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:15.680987 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.682104 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.682107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:15.683037 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:15.683047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.683665 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.683668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:15.683819 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:15.684512 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:15.684530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:15.686005 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:15.687549 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:15.687555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:15.687571 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:15.687898 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:15.688138 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:15.688141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:15.688143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:15.688271 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:15.688626 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:15.688632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:15.688881 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 15:17:15.689044 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:17:15.689359 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":136,"threads_started":1}
I20260927 15:17:15.691407 18725 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34307
I20260927 15:17:15.691701 18725 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:15.692279 18725 heartbeater.cc:511] Master 127.9.242.190:34307 requested a full tablet report, sending...
I20260927 15:17:15.693375 18400 ts_manager.cc:194] Registered new tserver with Master: 40e1d55a27b6460fa6cad54565c0f4a8 (127.9.242.130:42413)
I20260927 15:17:15.694144 18400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:41637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:17:15.702674 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34307
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:15.811615 18730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:15.811895 18730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:15.811956 18730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:15.816121 18730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:15.816191 18730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:15.816272 18730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:15.822088 18730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:15.823201 18730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:15.824257 18730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:15.833833 18736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:15.833848 18735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:15.833861 18738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:15.833972 18730 server_base.cc:1034] running on GCE node
I20260927 15:17:15.834450 18730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:15.835086 18730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:15.836249 18730 hybrid_clock.cc:648] HybridClock initialized: now 1790522235836227 us; error 35 us; skew 500 ppm
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522235, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:15.838912 18730 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:17:15.839798 18730 webserver.cc:460] Webserver started at http://127.9.242.131:36111/ using document root <none> and password file <none>
I20260927 15:17:15.840319 18730 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:15.840384 18730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:15.840572 18730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:15.842671 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "cb65c61da2864c59b065bde59a1c30c3"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "25599ed815074c482d3d93f83fba72a3"
server_key_iv: "8e629b023101bb278d65019c6b3af7cd"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.843191 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "cb65c61da2864c59b065bde59a1c30c3"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "25599ed815074c482d3d93f83fba72a3"
server_key_iv: "8e629b023101bb278d65019c6b3af7cd"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.846890 18730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:17:15.849325 18745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.850538 18730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:15.850632 18730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "cb65c61da2864c59b065bde59a1c30c3"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "25599ed815074c482d3d93f83fba72a3"
server_key_iv: "8e629b023101bb278d65019c6b3af7cd"
server_key_version: "encryptionkey@0"
I20260927 15:17:15.850756 18730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:15.877440 18730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:15.880539 18730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:15.880720 18730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:15.881296 18730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:15.882244 18730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:15.882302 18730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.882357 18730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:15.882390 18730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:15.890690 18730 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:43615
I20260927 15:17:15.890791 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:43615 every 8 connection(s)
I20260927 15:17:15.891600 18730 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:15.898842 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 18730
I20260927 15:17:15.898941 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:15.899232 10186 external_mini_cluster.cc:1428] Setting key 0f73b4f23f2d66620717b9d215905889
Sep 27 15:17:15 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522235, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:15.905048 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:15.893403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:50387 (local address 127.9.242.190:34307)
0927 15:17:15.893555 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:15.893559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:15.894496 (+ 937us) server_negotiation.cc:407] Connection header received
0927 15:17:15.895814 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:15.895817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:15.895873 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:15.895942 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:15.897102 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.897621 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.898745 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.898970 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.899077 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:15.901876 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:15.901879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:15.901881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:15.901903 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:15.903129 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.903619 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.903621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.903622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.903700 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:15.904083 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:15.904086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:15.904087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:15.904267 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:15.904351 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:15.904837 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:17:15.904931 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 15:17:15.905937 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:15.893839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:50387)
0927 15:17:15.894344 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:15.894387 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:15.895604 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:15.896126 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:15.896133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:15.896490 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:15.896962 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:15.896975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.897739 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.897742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:15.898609 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:15.898620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:15.899128 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:15.899133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:15.899323 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:15.899918 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:15.899933 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:15.901741 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:15.903232 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:15.903237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:15.903248 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:15.903492 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:15.903800 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:15.903803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:15.903805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:15.903967 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:15.904387 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:15.904393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:15.904645 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 15:17:15.904805 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:17:15.905060 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":122,"threads_started":1}
I20260927 15:17:15.906991 18859 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34307
I20260927 15:17:15.907207 18859 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:15.907652 18859 heartbeater.cc:511] Master 127.9.242.190:34307 requested a full tablet report, sending...
I20260927 15:17:15.908609 18400 ts_manager.cc:194] Registered new tserver with Master: cb65c61da2864c59b065bde59a1c30c3 (127.9.242.131:43615)
I20260927 15:17:15.909142 18400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:50387
I20260927 15:17:15.912845 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab.
Entry for principal oryx/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab.
WARNING: no policy specified for oryx/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab.
Entry for principal oryx/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab.
WARNING: no policy specified for oryx/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab.
Entry for principal oryx/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab.
I20260927 15:17:15.979888 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 18462
I20260927 15:17:15.984354 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 18596
I20260927 15:17:15.988867 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 18730
I20260927 15:17:15.992985 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 18368
I20260927 15:17:15.998073 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34307
--webserver_interface=127.9.242.190
--webserver_port=41785
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:34307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:16.105422 18868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:16.105711 18868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:16.105772 18868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:16.109803 18868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:16.109861 18868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:16.109879 18868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:16.109896 18868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:16.109925 18868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:16.115324 18868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:34307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=41785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:16.116390 18868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:16.117388 18868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:16.125555 18873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:16.125603 18874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:16.125559 18876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:16.125918 18868 server_base.cc:1034] running on GCE node
I20260927 15:17:16.126449 18868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:16.127602 18868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:16.128772 18868 hybrid_clock.cc:648] HybridClock initialized: now 1790522236128751 us; error 35 us; skew 500 ppm
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522236, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:16.131546 18868 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:17:16.132373 18868 webserver.cc:460] Webserver started at http://127.9.242.190:41785/ using document root <none> and password file <none>
I20260927 15:17:16.132893 18868 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:16.132987 18868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:16.136896 18868 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:16.139202 18883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.140398 18868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:16.140513 18868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "a3672cf5d1444942abb8b1fd4bacc0e1"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "c594a525fa84cdbd454bb7e95e130637"
server_key_iv: "c6a450541bce7e605c8121cce24f4f45"
server_key_version: "encryptionkey@0"
I20260927 15:17:16.141029 18868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:16.155112 18868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:16.157914 18868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:16.158092 18868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:16.165104 18868 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:34307
I20260927 15:17:16.165130 18935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:34307 every 8 connection(s)
I20260927 15:17:16.166288 18868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:16.170565 18936 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:17:16.172312 18936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Bootstrap starting.
I20260927 15:17:16.173866 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 18868
I20260927 15:17:16.174326 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:39797
--local_ip_for_outbound_sockets=127.9.242.129
--tserver_master_addrs=127.9.242.190:34307
--webserver_port=46199
--webserver_interface=127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:16.188858 18936 log.cc:824] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:16.195294 18936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:17:16.195626 18936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Bootstrap complete.
I20260927 15:17:16.199676 18936 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:16.200059 18936 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3672cf5d1444942abb8b1fd4bacc0e1, State: Initialized, Role: FOLLOWER
I20260927 15:17:16.200587 18936 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:16.200711 18936 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:16.200776 18936 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:16.200867 18936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:17:16.202805 18936 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:16.203107 18936 leader_election.cc:302] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3672cf5d1444942abb8b1fd4bacc0e1; no voters:
I20260927 15:17:16.203400 18936 leader_election.cc:288] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:17:16.203510 18940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:17:16.203732 18940 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [term 2 LEADER]: Becoming Leader. State: Replica: a3672cf5d1444942abb8b1fd4bacc0e1, State: Running, Role: LEADER
I20260927 15:17:16.204114 18940 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } }
I20260927 15:17:16.204663 18936 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:16.205376 18942 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3672cf5d1444942abb8b1fd4bacc0e1. Latest consensus state: current_term: 2 leader_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } } }
I20260927 15:17:16.205494 18942 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:16.206175 18941 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3672cf5d1444942abb8b1fd4bacc0e1" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34307 } } }
I20260927 15:17:16.206261 18941 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:16.206737 18948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:16.209439 18948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:16.210008 18948 catalog_manager.cc:1259] Loaded cluster ID: 012b05f5a5dc469587caf925a1ee198a
I20260927 15:17:16.210055 18948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:16.211669 18948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:16.212210 18948 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a3672cf5d1444942abb8b1fd4bacc0e1: Loaded TSK: 0
I20260927 15:17:16.213142 18948 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:17:16.298808 18938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:16.299055 18938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:16.299096 18938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:16.303042 18938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:16.303097 18938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:16.303154 18938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:16.308420 18938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.129.keytab
--principal=oryx/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:39797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=46199
--tserver_master_addrs=127.9.242.190:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:16.309343 18938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:16.310335 18938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:16.320319 18963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:16.320322 18964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:16.320626 18966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:16.320608 18938 server_base.cc:1034] running on GCE node
I20260927 15:17:16.321085 18938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:16.321933 18938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:16.323128 18938 hybrid_clock.cc:648] HybridClock initialized: now 1790522236323103 us; error 39 us; skew 500 ppm
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522236, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:16.326610 18938 init.cc:377] Logged in from keytab as oryx/127.9.242.129@KRBTEST.COM (short username oryx)
I20260927 15:17:16.327781 18938 webserver.cc:460] Webserver started at http://127.9.242.129:46199/ using document root <none> and password file <none>
I20260927 15:17:16.328488 18938 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:16.328595 18938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:16.333739 18938 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:16.336302 18973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.337810 18938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:17:16.337930 18938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "f626a1751e7f443e88673fb0f6ac76f8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "bbea2f92e10c73b9ec94a984f3a26797"
server_key_iv: "d61ba3a9b5fc1a7d92d740ad8adf56e6"
server_key_version: "encryptionkey@0"
I20260927 15:17:16.338634 18938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:16.355168 18938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:16.359061 18938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:16.359246 18938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:16.359928 18938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:16.360877 18938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:16.360931 18938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.360987 18938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:16.361016 18938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.371014 18938 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:39797
I20260927 15:17:16.371042 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:39797 every 8 connection(s)
I20260927 15:17:16.372074 18938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:16.380822 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 18938
I20260927 15:17:16.381327 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:42413
--local_ip_for_outbound_sockets=127.9.242.130
--tserver_master_addrs=127.9.242.190:34307
--webserver_port=34511
--webserver_interface=127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.384995 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.374270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:53537)
0927 15:17:16.374940 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.374990 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.375792 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.376855 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.376861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.377217 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.377704 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.377723 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.379053 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.379056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.379940 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.379949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.380757 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.380762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.380950 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.381640 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.381693 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.384245 (+ 2552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":111,"threads_started":1}
W20260927 15:17:16.385437 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.374177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:53537 (local address 127.9.242.190:34307)
0927 15:17:16.374576 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.374592 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.375125 (+ 533us) server_negotiation.cc:407] Connection header received
0927 15:17:16.376187 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.376199 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.376501 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.376775 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.377888 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.378916 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.380082 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.380573 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.380692 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.384523 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:53537: BlockingRecv error: recv got EOF from 127.9.242.129:53537 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
W20260927 15:17:16.385571 19087 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:34307 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.242.190:34307: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.390738 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.386074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:42551)
0927 15:17:16.386229 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.386243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.386320 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.386532 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.386535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.386689 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.386774 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.386779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.387460 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.387464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.388003 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.388011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.388409 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.388413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.388496 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.388973 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.388984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.390573 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 15:17:16.390818 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.386170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:42551 (local address 127.9.242.190:34307)
0927 15:17:16.386283 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.386287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.386300 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:16.386402 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.386405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.386448 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.386511 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.386873 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.387368 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.388101 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.388319 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.388401 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.390717 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:42551: BlockingRecv error: recv got EOF from 127.9.242.129:42551 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.395815 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.391270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:52037)
0927 15:17:16.391415 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.391428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.391501 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.391708 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.391711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.391847 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.391915 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.391920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.392588 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.392592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.393103 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.393110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.393493 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.393496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.393566 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.394077 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.394088 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.395682 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 15:17:16.395907 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.391349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:52037 (local address 127.9.242.190:34307)
0927 15:17:16.391455 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.391459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.391472 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:16.391579 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.391582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.391622 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.391681 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.392008 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.392496 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.393203 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.393405 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.393487 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.395807 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:52037: BlockingRecv error: recv got EOF from 127.9.242.129:52037 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 15:17:16.396095 19087 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:17:16.490432 19091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:16.490691 19091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:16.490732 19091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:16.494673 19091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:16.494740 19091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:16.494798 19091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:16.499964 19091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.130.keytab
--principal=oryx/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:42413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=34511
--tserver_master_addrs=127.9.242.190:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:16.500941 19091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:16.501973 19091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:16.511169 19099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:16.511193 19097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:16.511303 19091 server_base.cc:1034] running on GCE node
W20260927 15:17:16.511232 19096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:16.511750 19091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:16.512319 19091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:16.513478 19091 hybrid_clock.cc:648] HybridClock initialized: now 1790522236513453 us; error 40 us; skew 500 ppm
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522236, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:16.516031 19091 init.cc:377] Logged in from keytab as oryx/127.9.242.130@KRBTEST.COM (short username oryx)
I20260927 15:17:16.516861 19091 webserver.cc:460] Webserver started at http://127.9.242.130:34511/ using document root <none> and password file <none>
I20260927 15:17:16.517349 19091 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:16.517412 19091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:16.521023 19091 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:17:16.522974 19106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.524116 19091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:16.524204 19091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "40e1d55a27b6460fa6cad54565c0f4a8"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "2b4cd1d496f611e3cd339a38497c0aa9"
server_key_iv: "f717bbad8f5b7c5557b9443af59a67d8"
server_key_version: "encryptionkey@0"
I20260927 15:17:16.524708 19091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:16.553341 19091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:16.556278 19091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:16.556442 19091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:16.557027 19091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:16.557874 19091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:16.557919 19091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.557956 19091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:16.557969 19091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.566416 19091 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:42413
I20260927 15:17:16.566457 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:42413 every 8 connection(s)
I20260927 15:17:16.567569 19091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.577864 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 19091
W20260927 15:17:16.578131 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.569296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:47425 (local address 127.9.242.190:34307)
0927 15:17:16.569459 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.569463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.570491 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 15:17:16.571341 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.571344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.571400 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.571472 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.572661 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.573199 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.574380 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.574603 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.574691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.578004 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:47425: BlockingRecv error: recv got EOF from 127.9.242.130:47425 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:17:16.578536 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:43615
--local_ip_for_outbound_sockets=127.9.242.131
--tserver_master_addrs=127.9.242.190:34307
--webserver_port=36111
--webserver_interface=127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:17:16.578686 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.569719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:47425)
0927 15:17:16.570342 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.570381 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.571150 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.571659 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.571666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.572002 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.572527 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.572540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.573335 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.573338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.574264 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.574274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.574736 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.574740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.574899 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.575426 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.575442 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.577840 (+ 2398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":107,"threads_started":1}
W20260927 15:17:16.579166 19220 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:34307 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.242.190:34307: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.584316 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.579563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:60343)
0927 15:17:16.579670 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.579683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.579756 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.579958 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.579961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.580106 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.580197 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.580202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.580876 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.580879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.581408 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.581415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.581844 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.581847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.581930 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.582440 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.582452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.584158 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:17:16.584396 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.579667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:60343 (local address 127.9.242.190:34307)
0927 15:17:16.579776 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.579780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.579792 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:17:16.579831 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.579834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.579871 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.579931 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.580296 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.580785 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.581506 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.581746 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.581826 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.584295 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:60343: BlockingRecv error: recv got EOF from 127.9.242.130:60343 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.589692 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.584914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:54251)
0927 15:17:16.585063 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.585073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.585132 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.585370 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.585372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.585484 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.585551 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.585555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.586495 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.586497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.586996 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.587002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.587507 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.587509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.587559 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.587987 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.587996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.589580 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 15:17:16.589859 19220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:17:16.590026 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.584984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:54251 (local address 127.9.242.190:34307)
0927 15:17:16.585095 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.585099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.585117 (+ 18us) server_negotiation.cc:407] Connection header received
0927 15:17:16.585226 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.585229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.585274 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.585365 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.585641 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.586401 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.587093 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.587413 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.587510 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.589735 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:54251: BlockingRecv error: recv got EOF from 127.9.242.130:54251 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 15:17:16.688397 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:16.688647 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:16.688687 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:16.692625 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:16.692688 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:16.692744 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:16.697858 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/krb5kdc/oryx_127.9.242.131.keytab
--principal=oryx/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:43615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=36111
--tserver_master_addrs=127.9.242.190:34307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:16.698805 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:16.699712 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:16.709173 19228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:16.709173 19229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:16.709255 19223 server_base.cc:1034] running on GCE node
W20260927 15:17:16.709200 19231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:16.709796 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:16.710366 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:16.711520 19223 hybrid_clock.cc:648] HybridClock initialized: now 1790522236711508 us; error 26 us; skew 500 ppm
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522236, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:16.714246 19223 init.cc:377] Logged in from keytab as oryx/127.9.242.131@KRBTEST.COM (short username oryx)
I20260927 15:17:16.715088 19223 webserver.cc:460] Webserver started at http://127.9.242.131:36111/ using document root <none> and password file <none>
I20260927 15:17:16.715590 19223 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:16.715658 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:16.719383 19223 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 15:17:16.721473 19238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.722846 19223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 15:17:16.722949 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "cb65c61da2864c59b065bde59a1c30c3"
format_stamp: "Formatted at 2026-09-27 15:17:15 on dist-test-slave-m4lt"
server_key: "25599ed815074c482d3d93f83fba72a3"
server_key_iv: "8e629b023101bb278d65019c6b3af7cd"
server_key_version: "encryptionkey@0"
I20260927 15:17:16.723471 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:16.745941 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:16.748827 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:16.748996 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:16.749557 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:16.750396 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:16.750439 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.750480 19223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:16.750509 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:16.758997 19223 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:43615
I20260927 15:17:16.759037 19351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:43615 every 8 connection(s)
I20260927 15:17:16.759975 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:16.764889 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 19223
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 15:17:16.770689 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.761683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:57801 (local address 127.9.242.190:34307)
0927 15:17:16.761820 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.761824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.762845 (+ 1021us) server_negotiation.cc:407] Connection header received
0927 15:17:16.763683 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.763685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.763731 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.763791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.765078 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.765640 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.767175 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.767385 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.767466 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.770569 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:57801: BlockingRecv error: recv got EOF from 127.9.242.131:57801 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:16.771322 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.762069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:57801)
0927 15:17:16.762705 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.762746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.763478 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.764031 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.764039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.764454 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.764926 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.764939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.765815 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.765818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.767057 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.767067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.767512 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.767516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.767687 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.768247 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.768265 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.770462 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":101,"threads_started":1}
W20260927 15:17:16.771801 19352 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:34307 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.242.190:34307: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
I20260927 15:17:16.774124 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.765939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.9.242.190:34307)
0927 15:17:16.766153 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.766156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.766167 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:17:16.766197 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.766198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.766232 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.766299 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.767126 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.767633 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.768658 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.768870 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.768956 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.770307 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:16.770312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:16.770337 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:16.770367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:16.772130 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:16.772564 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:16.772570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:16.772576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:16.772728 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:16.772971 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:16.772974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:16.772976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:16.773300 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:16.773442 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:16.773740 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 15:17:16.773963 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.777019 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.772192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:36345)
0927 15:17:16.772336 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.772347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.772402 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.772594 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.772597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.772760 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.772861 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.772866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.773574 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.773577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.774349 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.774357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.774748 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.774751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.774834 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.775253 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.775264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.776931 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 15:17:16.777119 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.772256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:36345 (local address 127.9.242.190:34307)
0927 15:17:16.772383 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.772387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.772400 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:16.772465 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.772467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.772507 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.772581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.772949 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.773464 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.774450 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.774662 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.774740 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.777032 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:36345: BlockingRecv error: recv got EOF from 127.9.242.131:36345 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:16 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:16.781723 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:16.777462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:42161)
0927 15:17:16.777588 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:16.777602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:16.777702 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:16.777864 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:16.777866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:16.777973 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:16.778040 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.778044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.778668 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.778670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.779183 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:16.779190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.779609 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.779612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:16.779675 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.780091 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:16.780104 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:16.781599 (+ 1495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:17:16.781915 19352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:17:16.782040 19089 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:16.777533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:42161 (local address 127.9.242.190:34307)
0927 15:17:16.777679 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:16.777683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:16.777696 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:16.777767 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:16.777769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:16.777800 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:16.777854 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:16.778115 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.778586 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.779267 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:16.779457 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:16.779525 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:16.781763 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:42161: BlockingRecv error: recv got EOF from 127.9.242.131:42161 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 15:17:17 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:17 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:17.402125 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:17.396845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:38239)
0927 15:17:17.397199 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:17.397221 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:17.397336 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:17.397596 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:17.397599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:17.397834 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:17.397951 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:17.397959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.398711 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.398715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:17.399316 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:17.399325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.399743 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.399746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:17.399828 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:17.400268 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:17.400279 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:17.401971 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":172,"threads_started":1}
W20260927 15:17:17.402222 19358 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:17.396945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:38239 (local address 127.9.242.190:34307)
0927 15:17:17.397185 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:17.397189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:17.397308 (+ 119us) server_negotiation.cc:407] Connection header received
0927 15:17:17.397441 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:17.397443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:17.397487 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:17.397562 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:17.398064 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.398617 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.399426 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.399644 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.399731 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:17.402107 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:38239: BlockingRecv error: recv got EOF from 127.9.242.129:38239 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":62,"threads_started":1}
Sep 27 15:17:17 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:17 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:17.595499 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:17.590468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:60483)
0927 15:17:17.590733 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:17.590748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:17.590856 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:17.591089 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:17.591092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:17.591302 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:17.591395 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:17.591401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.592117 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.592120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:17.592685 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:17.592693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.593113 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.593117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:17.593204 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:17.593692 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:17.593703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:17.595345 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
W20260927 15:17:17.595583 19358 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:17.590543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:60483 (local address 127.9.242.190:34307)
0927 15:17:17.590706 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:17.590711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:17.590817 (+ 106us) server_negotiation.cc:407] Connection header received
0927 15:17:17.590941 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:17.590944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:17.590989 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:17.591054 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:17.591507 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.592018 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.592795 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.593015 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.593098 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:17.595482 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:60483: BlockingRecv error: recv got EOF from 127.9.242.130:60483 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 15:17:17 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:17 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:17.787806 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:17.782626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:47985)
0927 15:17:17.782861 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:17.782873 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:17.782988 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:17.783230 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:17.783233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:17.783418 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:17.783530 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:17.783537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.784306 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.784310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:17.784885 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:17.784893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.785353 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.785357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:17.785453 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:17.785939 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:17.785951 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:17.787658 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
W20260927 15:17:17.787994 19358 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:17.782648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:47985 (local address 127.9.242.190:34307)
0927 15:17:17.782784 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:17.782789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:17.782951 (+ 162us) server_negotiation.cc:407] Connection header received
0927 15:17:17.783077 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:17.783080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:17.783129 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:17.783191 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:17.783646 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.784196 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.784994 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:17.785247 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:17.785336 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:17.787793 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:47985: BlockingRecv error: recv got EOF from 127.9.242.131:47985 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:17:18 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:18 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:18.408350 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:18.402903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:45763)
0927 15:17:18.403168 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:18.403183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:18.403285 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:18.403602 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:18.403605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:18.403803 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:18.403915 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:18.403922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.404909 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.404912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:18.405501 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:18.405511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.406034 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.406038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:18.406103 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:18.406600 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:18.406612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:18.408216 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
W20260927 15:17:18.408456 19362 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:18.403047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:45763 (local address 127.9.242.190:34307)
0927 15:17:18.403321 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:18.403325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:18.403340 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:17:18.403381 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:18.403384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:18.403443 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:18.403533 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:18.404103 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.404804 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.405623 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.405929 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.406023 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:18.408346 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:45763: BlockingRecv error: recv got EOF from 127.9.242.129:45763 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
Sep 27 15:17:18 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:18 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:18.601348 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:18.596299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:35289)
0927 15:17:18.596563 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:18.596575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:18.596674 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:18.596926 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:18.596929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:18.597094 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:18.597192 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:18.597198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.597941 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.597943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:18.598486 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:18.598493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.598901 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.598903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:18.598958 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:18.599405 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:18.599416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:18.601178 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
W20260927 15:17:18.601418 19362 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:18.596447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:35289 (local address 127.9.242.190:34307)
0927 15:17:18.596575 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:18.596579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:18.596628 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:17:18.596738 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:18.596740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:18.596780 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:18.596854 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:18.597318 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.597865 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.598591 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.598817 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.598891 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:18.601302 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:35289: BlockingRecv error: recv got EOF from 127.9.242.130:35289 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 15:17:18 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:18 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:18.793375 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:18.788492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:58455)
0927 15:17:18.788793 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:18.788806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:18.788901 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:18.789085 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:18.789087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:18.789247 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:18.789337 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:18.789342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.790078 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.790081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:18.790620 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:18.790627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.791013 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.791015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:18.791070 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:18.791545 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:18.791556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:18.793247 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
W20260927 15:17:18.793483 19362 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:18.788638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:58455 (local address 127.9.242.190:34307)
0927 15:17:18.788771 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:18.788775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:18.788863 (+ 88us) server_negotiation.cc:407] Connection header received
0927 15:17:18.788962 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:18.788964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:18.789006 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:18.789071 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:18.789415 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.789996 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.790721 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:18.790926 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:18.791004 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:18.793383 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:58455: BlockingRecv error: recv got EOF from 127.9.242.131:58455 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:17:19 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:19 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:19.413897 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:19.409049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:45485)
0927 15:17:19.409298 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:19.409310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:19.409395 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:19.409625 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:19.409627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:19.409807 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:19.409899 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:19.409904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.410626 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.410628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:19.411175 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:19.411182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.411560 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.411562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:19.411615 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:19.412043 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:19.412053 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:19.413748 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
W20260927 15:17:19.414009 19366 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:19.409114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:45485 (local address 127.9.242.190:34307)
0927 15:17:19.409345 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:19.409349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:19.409362 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:19.409459 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:19.409461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:19.409502 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:19.409592 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:19.409989 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.410512 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.411265 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.411481 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.411581 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:19.413880 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:45485: BlockingRecv error: recv got EOF from 127.9.242.129:45485 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
Sep 27 15:17:19 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:19 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:19.607456 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:19.602191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:47441)
0927 15:17:19.602466 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:19.602482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:19.602581 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:19.602823 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:19.602826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:19.603007 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:19.603113 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:19.603120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.603850 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.603853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:19.604457 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:19.604466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.604876 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.604879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:19.604956 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:19.605492 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:19.605504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:19.607307 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
W20260927 15:17:19.607538 19366 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:19.602308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:47441 (local address 127.9.242.190:34307)
0927 15:17:19.602453 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:19.602458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:19.602552 (+ 94us) server_negotiation.cc:407] Connection header received
0927 15:17:19.602661 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:19.602664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:19.602708 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:19.602782 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:19.603226 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.603750 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.604556 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.604774 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.604865 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:19.607440 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:47441: BlockingRecv error: recv got EOF from 127.9.242.130:47441 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 15:17:19 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:19 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:19.799811 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:19.794169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:47671)
0927 15:17:19.794454 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:19.794467 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:19.794573 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:19.794817 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:19.794819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:19.794986 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:19.795140 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:19.795145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.795859 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.795862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:19.796481 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:19.796491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.796913 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.796915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:19.796972 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:19.797519 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:19.797532 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:19.799646 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
W20260927 15:17:19.799865 19366 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:19.794280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:47671 (local address 127.9.242.190:34307)
0927 15:17:19.794403 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:19.794407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:19.794533 (+ 126us) server_negotiation.cc:407] Connection header received
0927 15:17:19.794679 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:19.794681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:19.794730 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:19.794791 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:19.795234 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.795759 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.796596 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:19.796827 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:19.796904 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:19.799772 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:47671: BlockingRecv error: recv got EOF from 127.9.242.131:47671 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 15:17:20 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:20 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:20.424870 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:20.414685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:46091)
0927 15:17:20.414933 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:20.414946 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:20.415032 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:20.415248 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:20.415250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:20.415411 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:20.415503 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:20.415508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.416318 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.416320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:20.416856 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:20.416863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.417327 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.417330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:20.417388 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:20.422969 (+ 5581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:20.422982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:20.424750 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
W20260927 15:17:20.425024 19370 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:20.414761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:46091 (local address 127.9.242.190:34307)
0927 15:17:20.414995 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:20.414998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:20.415013 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:17:20.415107 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:20.415109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:20.415150 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:20.415223 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:20.415649 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.416224 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.417014 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.417237 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.417315 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:20.424898 (+ 7583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:46091: BlockingRecv error: recv got EOF from 127.9.242.129:46091 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
Sep 27 15:17:20 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:20 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:20.613523 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:20.608229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:60575)
0927 15:17:20.608543 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:20.608555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:20.608658 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:20.608889 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:20.608892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:20.609153 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:20.609246 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:20.609252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.610058 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.610062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:20.610607 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:20.610615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.611026 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.611030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:20.611120 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:20.611585 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:20.611596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:20.613375 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":70,"threads_started":1}
W20260927 15:17:20.613669 19370 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:20.608295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:60575 (local address 127.9.242.190:34307)
0927 15:17:20.608431 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:20.608435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:20.608624 (+ 189us) server_negotiation.cc:407] Connection header received
0927 15:17:20.608745 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:20.608748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:20.608796 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:20.608859 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:20.609357 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.609949 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.610711 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.610925 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.611006 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:20.613548 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:60575: BlockingRecv error: recv got EOF from 127.9.242.130:60575 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:17:20 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:20 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:20.809263 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:20.800608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:53043)
0927 15:17:20.800842 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:20.800856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:20.800957 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:20.801199 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:20.801201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:20.801367 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:20.801460 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:20.801464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.802291 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.802293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:20.802847 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:20.802854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.803365 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.803368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:20.803428 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:20.807382 (+ 3954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:20.807394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:20.809140 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
W20260927 15:17:20.809381 19370 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:20.800699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:53043 (local address 127.9.242.190:34307)
0927 15:17:20.800862 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:20.800866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:20.800911 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:17:20.801037 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:20.801040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:20.801085 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:20.801162 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:20.801569 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.802203 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.803029 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:20.803252 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:20.803361 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:20.809276 (+ 5915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:53043: BlockingRecv error: recv got EOF from 127.9.242.131:53043 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 15:17:21 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:21 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:21.430822 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:21.425709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:49577)
0927 15:17:21.425960 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:21.425974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:21.426053 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:21.426248 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:21.426250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:21.426417 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:21.426502 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:21.426507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.427343 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.427346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:21.427979 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:21.427988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.428417 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.428419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:21.428474 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:21.428955 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:21.428966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:21.430689 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
W20260927 15:17:21.431094 19374 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:21.425783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:49577 (local address 127.9.242.190:34307)
0927 15:17:21.426029 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:21.426032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:21.426045 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:21.426123 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:21.426125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:21.426163 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:21.426235 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:21.426617 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.427248 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.428074 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.428330 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.428412 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:21.430832 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:49577: BlockingRecv error: recv got EOF from 127.9.242.129:49577 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
Sep 27 15:17:21 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:21 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:21.619578 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:21.614325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:50235)
0927 15:17:21.614568 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:21.614584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:21.614691 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:21.614930 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:21.614932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:21.615117 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:21.615212 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:21.615218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.616012 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.616016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:21.616712 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:21.616724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.617201 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.617204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:21.617270 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:21.617789 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:21.617800 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:21.619454 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
W20260927 15:17:21.619818 19374 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:21.614502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:50235 (local address 127.9.242.190:34307)
0927 15:17:21.614627 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:21.614631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:21.614645 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:17:21.614775 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:21.614778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:21.614824 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:21.614894 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:21.615319 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.615889 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.616823 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.617106 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.617191 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:21.619606 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:50235: BlockingRecv error: recv got EOF from 127.9.242.130:50235 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:17:21 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:21 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:21.815076 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:21.810083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:49275)
0927 15:17:21.810313 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:21.810326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:21.810418 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:21.810625 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:21.810627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:21.810790 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:21.810883 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:21.810887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.811759 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.811762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:21.812329 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:21.812337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.812723 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.812726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:21.812787 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:21.813297 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:21.813309 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:21.814964 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
W20260927 15:17:21.815207 19374 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:21.810172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:49275 (local address 127.9.242.190:34307)
0927 15:17:21.810289 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:21.810293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:21.810396 (+ 103us) server_negotiation.cc:407] Connection header received
0927 15:17:21.810487 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:21.810489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:21.810532 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:21.810601 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:21.810966 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.811642 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.812412 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:21.812622 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:21.812703 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:21.815100 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:49275: BlockingRecv error: recv got EOF from 127.9.242.131:49275 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 15:17:22 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:22 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:22.436970 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:22.431751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:35777)
0927 15:17:22.432106 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:22.432119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:22.432237 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:22.432512 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:22.432515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:22.432714 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:22.432813 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:22.432818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.433527 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.433530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:22.434148 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:22.434157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.434567 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.434570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:22.434640 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:22.435122 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:22.435133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:22.436859 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":79,"threads_started":1}
W20260927 15:17:22.437098 19378 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:22.431790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:35777 (local address 127.9.242.190:34307)
0927 15:17:22.432019 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:22.432023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:22.432196 (+ 173us) server_negotiation.cc:407] Connection header received
0927 15:17:22.432336 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:22.432339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:22.432392 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:22.432465 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:22.432905 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.433430 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.434232 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.434462 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.434527 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:22.436986 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:35777: BlockingRecv error: recv got EOF from 127.9.242.129:35777 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
Sep 27 15:17:22 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:22 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:22.625195 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:22.620272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:35907)
0927 15:17:22.620513 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:22.620526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:22.620638 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:22.620889 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:22.620892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:22.621080 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:22.621171 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:22.621177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.621925 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.621928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:22.622472 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:22.622480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.622889 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.622892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:22.622970 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:22.623434 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:22.623445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:22.625051 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
W20260927 15:17:22.625305 19378 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:22.620361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:35907 (local address 127.9.242.190:34307)
0927 15:17:22.620503 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:22.620508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:22.620593 (+ 85us) server_negotiation.cc:407] Connection header received
0927 15:17:22.620747 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:22.620750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:22.620796 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:22.620858 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:22.621279 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.621832 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.622577 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.622795 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.622878 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:22.625209 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:35907: BlockingRecv error: recv got EOF from 127.9.242.130:35907 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:17:22 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:22 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:22.820734 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:22.815792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:33709)
0927 15:17:22.816047 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:22.816063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:22.816164 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:22.816366 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:22.816368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:22.816552 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:22.816643 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:22.816648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.817348 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.817351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:22.817975 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:22.817984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.818410 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.818413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:22.818469 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:22.818934 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:22.818945 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:22.820609 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":66,"threads_started":1}
W20260927 15:17:22.820845 19378 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:22.815893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:33709 (local address 127.9.242.190:34307)
0927 15:17:22.816033 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:22.816036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:22.816130 (+ 94us) server_negotiation.cc:407] Connection header received
0927 15:17:22.816232 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:22.816234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:22.816278 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:22.816340 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:22.816737 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.817261 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.818093 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:22.818323 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:22.818401 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:22.820730 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:33709: BlockingRecv error: recv got EOF from 127.9.242.131:33709 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 15:17:23 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:23 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:23.443197 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:23.437847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:56101)
0927 15:17:23.438179 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:23.438192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:23.438287 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:23.438533 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:23.438535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:23.438711 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:23.438798 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:23.438804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.439557 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.439560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:23.440095 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:23.440102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.440570 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.440572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:23.440631 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:23.441111 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:23.441123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:23.443041 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
W20260927 15:17:23.443329 19382 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:23.437945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:56101 (local address 127.9.242.190:34307)
0927 15:17:23.438234 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:23.438238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:23.438251 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:23.438365 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:23.438367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:23.438409 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:23.438512 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:23.438895 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.439441 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.440216 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.440468 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.440550 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:23.443213 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:56101: BlockingRecv error: recv got EOF from 127.9.242.129:56101 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
Sep 27 15:17:23 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:23 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:23.631428 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:23.626073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:32813)
0927 15:17:23.626373 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:23.626388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:23.626484 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:23.626731 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:23.626734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:23.626981 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:23.627088 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:23.627096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.627996 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.627998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:23.628636 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:23.628645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.629048 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.629050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:23.629111 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:23.629562 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:23.629573 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:23.631315 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":76,"threads_started":1}
W20260927 15:17:23.631599 19382 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:23.626105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:32813 (local address 127.9.242.190:34307)
0927 15:17:23.626251 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:23.626255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:23.626455 (+ 200us) server_negotiation.cc:407] Connection header received
0927 15:17:23.626575 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:23.626579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:23.626621 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:23.626694 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:23.627234 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.627910 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.628745 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.628961 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.629044 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:23.631442 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:32813: BlockingRecv error: recv got EOF from 127.9.242.130:32813 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 15:17:23 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:23 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:23.826900 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:23.821464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:43991)
0927 15:17:23.821814 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:23.821828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:23.821927 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:23.822179 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:23.822182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:23.822348 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:23.822444 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:23.822459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.823210 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.823214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:23.823810 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:23.823820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.824237 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.824240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:23.824303 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:23.824796 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:23.824811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:23.826744 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":92,"threads_started":1}
W20260927 15:17:23.827118 19382 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:23.821552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:43991 (local address 127.9.242.190:34307)
0927 15:17:23.821731 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:23.821736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:23.821892 (+ 156us) server_negotiation.cc:407] Connection header received
0927 15:17:23.822005 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:23.822007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:23.822052 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:23.822117 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:23.822558 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.823088 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.823916 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:23.824140 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:23.824218 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:23.826917 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:43991: BlockingRecv error: recv got EOF from 127.9.242.131:43991 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:17:24 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:24 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:24.449033 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:24.443941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:38185)
0927 15:17:24.444221 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:24.444235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:24.444320 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:24.444528 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:24.444530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:24.444739 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:24.444836 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:24.444842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.445647 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.445679 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:24.446241 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:24.446249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.446649 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.446652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:24.446706 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:24.447202 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:24.447214 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:24.448886 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
W20260927 15:17:24.449245 19386 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:24.444067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:38185 (local address 127.9.242.190:34307)
0927 15:17:24.444288 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:24.444291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:24.444304 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:24.444399 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:24.444401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:24.444442 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:24.444506 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:24.444929 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.445554 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.446339 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.446564 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.446641 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:24.449017 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:38185: BlockingRecv error: recv got EOF from 127.9.242.129:38185 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
Sep 27 15:17:24 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:24 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:24.637737 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:24.632096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:39501)
0927 15:17:24.632366 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:24.632381 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:24.632505 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:24.632753 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:24.632755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:24.632947 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:24.633106 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:24.633112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.634089 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.634092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:24.634660 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:24.634668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.635077 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.635081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:24.635159 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:24.635638 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:24.635649 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:24.637539 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
W20260927 15:17:24.637817 19386 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:24.632207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:39501 (local address 127.9.242.190:34307)
0927 15:17:24.632364 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:24.632368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:24.632451 (+ 83us) server_negotiation.cc:407] Connection header received
0927 15:17:24.632591 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:24.632594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:24.632652 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:24.632716 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:24.633228 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.633978 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.634769 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.634983 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.635066 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:24.637715 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:39501: BlockingRecv error: recv got EOF from 127.9.242.130:39501 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 15:17:24 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:24 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:24.832922 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:24.827634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:52753)
0927 15:17:24.827939 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:24.827950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:24.828057 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:24.828284 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:24.828287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:24.828476 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:24.828590 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:24.828598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.829344 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.829348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:24.829942 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:24.829950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.830435 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.830438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:24.830520 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:24.831002 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:24.831016 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:24.832805 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":75,"threads_started":1}
W20260927 15:17:24.833073 19386 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:24.827708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:52753 (local address 127.9.242.190:34307)
0927 15:17:24.827839 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:24.827844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:24.828017 (+ 173us) server_negotiation.cc:407] Connection header received
0927 15:17:24.828139 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:24.828142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:24.828193 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:24.828256 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:24.828696 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.829241 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.830110 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:24.830327 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:24.830418 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:24.832938 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:52753: BlockingRecv error: recv got EOF from 127.9.242.131:52753 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 15:17:25 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:25 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:25.460824 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:25.449829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:60197)
0927 15:17:25.450149 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:25.450161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:25.450257 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:25.450510 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:25.450513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:25.450686 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:25.450785 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:25.450791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.451500 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.451503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:25.452070 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:25.452078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.452566 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.452569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:25.452639 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:25.458596 (+ 5957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:25.458611 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:25.460689 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":69,"threads_started":1}
W20260927 15:17:25.461076 19390 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:25.449911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:60197 (local address 127.9.242.190:34307)
0927 15:17:25.450121 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:25.450124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:25.450222 (+ 98us) server_negotiation.cc:407] Connection header received
0927 15:17:25.450323 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:25.450325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:25.450367 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:25.450441 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:25.450880 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.451399 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.452194 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.452424 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.452532 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:25.460841 (+ 8309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:60197: BlockingRecv error: recv got EOF from 127.9.242.129:60197 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":60,"threads_started":1}
Sep 27 15:17:25 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:25 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:25.644084 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:25.638467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:36167)
0927 15:17:25.638736 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:25.638757 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:25.638853 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:25.639128 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:25.639130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:25.639309 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:25.639406 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:25.639413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.640258 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.640261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:25.640819 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:25.640828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.641254 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.641257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:25.641324 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:25.641906 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:25.641918 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:25.643866 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
W20260927 15:17:25.644140 19390 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:25.638573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:36167 (local address 127.9.242.190:34307)
0927 15:17:25.638722 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:25.638726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:25.638847 (+ 121us) server_negotiation.cc:407] Connection header received
0927 15:17:25.638919 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:25.638921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:25.638969 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:25.639052 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:25.639547 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.640147 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.640925 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.641155 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.641231 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:25.644041 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:36167: BlockingRecv error: recv got EOF from 127.9.242.130:36167 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 15:17:25 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:25 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:25.839044 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:25.833688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:51259)
0927 15:17:25.834011 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:25.834026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:25.834129 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:25.834375 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:25.834378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:25.834595 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:25.834714 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:25.834721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.835491 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.835494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:25.836057 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:25.836065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.836480 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.836483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:25.836547 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:25.837017 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:25.837029 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:25.838914 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
W20260927 15:17:25.839246 19390 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:25.833729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:51259 (local address 127.9.242.190:34307)
0927 15:17:25.833857 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:25.833862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:25.834096 (+ 234us) server_negotiation.cc:407] Connection header received
0927 15:17:25.834208 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:25.834211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:25.834265 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:25.834349 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:25.834809 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.835392 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.836153 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:25.836381 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:25.836453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:25.839123 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:51259: BlockingRecv error: recv got EOF from 127.9.242.131:51259 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 15:17:26 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:26 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:26.466840 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:26.461587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:54099)
0927 15:17:26.461854 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:26.461871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:26.461958 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:26.462173 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:26.462175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:26.462344 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:26.462433 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:26.462438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.463202 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.463205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:26.463818 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:26.463829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.464332 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.464335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:26.464404 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:26.464928 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:26.464939 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:26.466723 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
W20260927 15:17:26.466979 19394 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:26.461716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:54099 (local address 127.9.242.190:34307)
0927 15:17:26.461956 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:26.461960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:26.461974 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:17:26.462029 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:26.462031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:26.462071 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:26.462141 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:26.462527 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.463076 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.463941 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.464209 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.464303 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:26.466857 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:54099: BlockingRecv error: recv got EOF from 127.9.242.129:54099 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
Sep 27 15:17:26 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:26 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:26.650285 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:26.644916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:50447)
0927 15:17:26.645193 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:26.645208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:26.645300 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:26.645560 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:26.645564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:26.645772 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:26.645870 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:26.645876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.646669 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.646672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:26.647240 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:26.647248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.647720 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.647723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:26.647790 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:26.648370 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:26.648383 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:26.650164 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":102,"threads_started":1}
W20260927 15:17:26.650422 19394 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:26.644966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:50447 (local address 127.9.242.190:34307)
0927 15:17:26.645100 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:26.645105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:26.645270 (+ 165us) server_negotiation.cc:407] Connection header received
0927 15:17:26.645372 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:26.645374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:26.645415 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:26.645482 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:26.645978 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.646565 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.647337 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.647599 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.647678 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:26.650310 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:50447: BlockingRecv error: recv got EOF from 127.9.242.130:50447 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 15:17:26 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:26 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:26.845341 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:26.839864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:54241)
0927 15:17:26.840133 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:26.840146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:26.840239 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:26.840467 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:26.840469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:26.840639 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:26.840732 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:26.840738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.841536 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.841538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:26.842198 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:26.842208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.842775 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.842779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:26.842868 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:26.843324 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:26.843335 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:26.845223 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":69,"threads_started":1}
W20260927 15:17:26.845503 19394 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:26.840035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:54241 (local address 127.9.242.190:34307)
0927 15:17:26.840154 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:26.840158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:26.840197 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:17:26.840312 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:26.840314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:26.840355 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:26.840427 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:26.840818 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.841409 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.842299 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:26.842648 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:26.842729 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:26.845374 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:54241: BlockingRecv error: recv got EOF from 127.9.242.131:54241 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:17:27 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:27 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:27.473448 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:27.467668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:52713)
0927 15:17:27.467931 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:27.467944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:27.468040 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:27.468301 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:27.468303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:27.468522 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:27.468625 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:27.468632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.469600 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.469604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:27.470296 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:27.470306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.470800 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.470804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:27.470876 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:27.471408 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:27.471420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:27.473284 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260927 15:17:27.473578 19398 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:27.467758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:52713 (local address 127.9.242.190:34307)
0927 15:17:27.468013 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:27.468018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:27.468037 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:17:27.468129 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:27.468133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:27.468185 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:27.468283 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:27.468747 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.469492 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.470446 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.470669 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.470761 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:27.473422 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:52713: BlockingRecv error: recv got EOF from 127.9.242.129:52713 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
Sep 27 15:17:27 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:27 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:27.656857 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:27.651104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:35939)
0927 15:17:27.651364 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:27.651377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:27.651480 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:27.651838 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:27.651841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:27.652107 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:27.652202 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:27.652208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.652998 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.653001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:27.653566 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:27.653574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.654125 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.654129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:27.654197 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:27.654660 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:27.654671 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:27.656738 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
W20260927 15:17:27.656950 19398 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:27.651190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:35939 (local address 127.9.242.190:34307)
0927 15:17:27.651338 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:27.651349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:27.651443 (+ 94us) server_negotiation.cc:407] Connection header received
0927 15:17:27.651586 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:27.651589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:27.651637 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:27.651705 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:27.652307 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.652852 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.653703 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.653929 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.654007 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:27.656856 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:35939: BlockingRecv error: recv got EOF from 127.9.242.130:35939 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 15:17:27 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:27 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:27.851343 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:27.846111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:36445)
0927 15:17:27.846383 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:27.846396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:27.846533 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:27.846736 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:27.846739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:27.846917 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:27.847033 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:27.847039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.847782 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.847785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:27.848351 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:27.848359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.848752 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.848754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:27.848811 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:27.849282 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:27.849293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:27.851214 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
W20260927 15:17:27.851449 19398 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:27.846189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:36445 (local address 127.9.242.190:34307)
0927 15:17:27.846320 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:27.846324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:27.846517 (+ 193us) server_negotiation.cc:407] Connection header received
0927 15:17:27.846599 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:27.846601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:27.846649 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:27.846713 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:27.847133 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.847672 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.848438 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:27.848667 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:27.848742 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:27.851349 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:36445: BlockingRecv error: recv got EOF from 127.9.242.131:36445 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 15:17:28 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:28 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:28.479274 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:28.474174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:59749)
0927 15:17:28.474431 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:28.474445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:28.474537 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:28.474752 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:28.474755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:28.474919 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:28.475009 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:28.475014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.475751 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.475754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:28.476347 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:28.476357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.476797 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.476799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:28.476859 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:28.477361 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:28.477373 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:28.479159 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
W20260927 15:17:28.479423 19402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:28.474246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:59749 (local address 127.9.242.190:34307)
0927 15:17:28.474454 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:28.474458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:28.474492 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:17:28.474609 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:28.474611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:28.474656 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:28.474727 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:28.475123 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.475649 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.476455 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.476711 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.476788 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:28.479298 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:59749: BlockingRecv error: recv got EOF from 127.9.242.129:59749 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
Sep 27 15:17:28 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:28 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:28.663090 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:28.657742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:53617)
0927 15:17:28.658019 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:28.658032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:28.658136 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:28.658413 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:28.658416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:28.658593 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:28.658687 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:28.658692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.659549 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.659552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:28.660118 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:28.660126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.660547 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.660551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:28.660630 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:28.661135 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:28.661146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:28.662962 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
W20260927 15:17:28.663266 19402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:28.657827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:53617 (local address 127.9.242.190:34307)
0927 15:17:28.657969 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:28.657974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:28.658127 (+ 153us) server_negotiation.cc:407] Connection header received
0927 15:17:28.658224 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:28.658227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:28.658281 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:28.658348 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:28.658813 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.659419 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.660228 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.660444 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.660528 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:28.663094 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:53617: BlockingRecv error: recv got EOF from 127.9.242.130:53617 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 15:17:28 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:28 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:28.857690 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:28.852242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:51791)
0927 15:17:28.852518 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:28.852530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:28.852636 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:28.852858 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:28.852860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:28.853039 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:28.853183 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:28.853190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.853999 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.854002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:28.854567 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:28.854576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.855064 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.855068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:28.855141 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:28.855639 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:28.855652 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:28.857543 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
W20260927 15:17:28.857852 19402 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:28.852344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:51791 (local address 127.9.242.190:34307)
0927 15:17:28.852470 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:28.852474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:28.852601 (+ 127us) server_negotiation.cc:407] Connection header received
0927 15:17:28.852713 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:28.852715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:28.852761 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:28.852835 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:28.853280 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.853867 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.854671 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:28.854886 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:28.854962 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:28.857729 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:51791: BlockingRecv error: recv got EOF from 127.9.242.131:51791 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:17:29 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:29 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:29.485203 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:29.480102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:41439)
0927 15:17:29.480346 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:29.480359 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:29.480445 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:29.480660 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:29.480662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:29.480833 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:29.480924 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:29.480929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.481704 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.481707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:29.482299 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:29.482308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.482698 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.482701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:29.482764 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:29.483258 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:29.483270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:29.485065 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
W20260927 15:17:29.485320 19406 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:29.480230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:41439 (local address 127.9.242.190:34307)
0927 15:17:29.480446 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:29.480449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:29.480462 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:29.480516 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:29.480518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:29.480564 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:29.480633 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:29.481019 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.481580 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.482388 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.482604 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.482669 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:29.485208 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:41439: BlockingRecv error: recv got EOF from 127.9.242.129:41439 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
Sep 27 15:17:29 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:29 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:29.669003 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:29.663760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:34195)
0927 15:17:29.664029 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:29.664042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:29.664133 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:29.664376 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:29.664379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:29.664591 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:29.664690 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:29.664696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.665368 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.665370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:29.666002 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:29.666011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.666425 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.666428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:29.666490 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:29.667001 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:29.667013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:29.668876 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
W20260927 15:17:29.669157 19406 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:29.663926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:34195 (local address 127.9.242.190:34307)
0927 15:17:29.664042 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:29.664051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:29.664088 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:17:29.664194 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:29.664196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:29.664236 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:29.664298 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:29.664781 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.665287 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.666124 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.666339 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.666410 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:29.669014 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:34195: BlockingRecv error: recv got EOF from 127.9.242.130:34195 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 15:17:29 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:29 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:29.864363 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:29.858660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:57531)
0927 15:17:29.858906 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:29.858919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:29.859043 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:29.859367 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:29.859370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:29.859598 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:29.859697 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:29.859704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.860552 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.860555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:29.861149 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:29.861157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.861614 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.861618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:29.861738 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:29.862306 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:29.862319 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:29.864239 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
W20260927 15:17:29.864493 19406 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:29.858776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:57531 (local address 127.9.242.190:34307)
0927 15:17:29.858882 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:29.858885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:29.859004 (+ 119us) server_negotiation.cc:407] Connection header received
0927 15:17:29.859150 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:29.859153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:29.859213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:29.859300 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:29.859824 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.860405 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.861265 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:29.861509 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:29.861598 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:29.864393 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:57531: BlockingRecv error: recv got EOF from 127.9.242.131:57531 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 27 15:17:30 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:30 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:30.495168 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:30.486093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:45163)
0927 15:17:30.486359 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:30.486372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:30.486460 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:30.486735 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:30.486737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:30.486907 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:30.486998 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:30.487003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.487871 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.487875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:30.488577 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:30.488588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.489285 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.489288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:30.489355 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:30.493100 (+ 3745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:30.493112 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:30.495031 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":92,"threads_started":1}
W20260927 15:17:30.495381 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:30.486197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:45163 (local address 127.9.242.190:34307)
0927 15:17:30.486516 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:30.486520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:30.486534 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:17:30.486584 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:30.486587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:30.486629 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:30.486713 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:30.487148 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.487731 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.488753 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.489168 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.489260 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:30.495237 (+ 5977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:45163: BlockingRecv error: recv got EOF from 127.9.242.129:45163 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
Sep 27 15:17:30 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:30 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:30.675305 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:30.669791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:60017)
0927 15:17:30.670130 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:30.670144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:30.670247 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:30.670471 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:30.670473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:30.670646 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:30.670739 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:30.670744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.671481 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.671484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:30.672077 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:30.672087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.672571 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.672576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:30.672672 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:30.673246 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:30.673258 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:30.675169 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
W20260927 15:17:30.675458 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:30.669878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:60017 (local address 127.9.242.190:34307)
0927 15:17:30.670022 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:30.670027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:30.670212 (+ 185us) server_negotiation.cc:407] Connection header received
0927 15:17:30.670328 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:30.670330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:30.670374 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:30.670431 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:30.670855 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.671382 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.672186 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.672436 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.672531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:30.675337 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:60017: BlockingRecv error: recv got EOF from 127.9.242.130:60017 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 15:17:30 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:30 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.131@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:30.871452 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:30.865219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.131:59003)
0927 15:17:30.865445 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:30.865460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:30.865577 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:30.865898 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:30.865901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:30.866143 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:30.866247 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:30.866253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.867312 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.867315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:30.868018 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:30.868030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.868613 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.868618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:30.868708 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:30.869239 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:30.869254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:30.871292 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":57,"threads_started":1}
W20260927 15:17:30.871555 19410 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:30.865263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:59003 (local address 127.9.242.190:34307)
0927 15:17:30.865375 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:30.865379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:30.865548 (+ 169us) server_negotiation.cc:407] Connection header received
0927 15:17:30.865698 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:30.865702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:30.865768 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:30.865853 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:30.866360 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.867154 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.868139 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:30.868479 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:30.868585 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:30.871448 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.131:59003: BlockingRecv error: recv got EOF from 127.9.242.131:59003 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 15:17:31 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:31 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.129@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:31.501053 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:31.496089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.129:46959)
0927 15:17:31.496350 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:31.496363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:31.496444 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:31.496653 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:31.496655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:31.496824 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:31.496915 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:31.496919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.497699 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.497702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:31.498349 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:31.498360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.498774 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.498777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:31.498842 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:31.499270 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:31.499281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:31.500933 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
W20260927 15:17:31.501240 19414 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:31.496234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:46959 (local address 127.9.242.190:34307)
0927 15:17:31.496436 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:31.496439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:31.496452 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:31.496515 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:31.496518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:31.496557 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:31.496626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:31.496995 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.497557 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.498453 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.498668 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.498741 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:31.501103 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.129:46959: BlockingRecv error: recv got EOF from 127.9.242.129:46959 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
Sep 27 15:17:31 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 15:17:31 dist-test-slave-m4lt krb5kdc[18355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.242.130@KRBTEST.COM for oryx/127.9.242.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 15:17:31.681864 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:31.676354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34307 (local address 127.9.242.130:48641)
0927 15:17:31.676682 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:31.676695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:31.676786 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:31.677044 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:31.677048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:31.677274 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:31.677387 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:31.677394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.678127 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.678130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:31.678741 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:31.678750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.679221 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.679223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:31.679285 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:31.679778 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:31.679789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:31.681740 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.242.190:34307: Server oryx/127.9.242.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":71,"threads_started":1}
W20260927 15:17:31.681955 19414 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:31.676417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:48641 (local address 127.9.242.190:34307)
0927 15:17:31.676540 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:31.676545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:31.676766 (+ 221us) server_negotiation.cc:407] Connection header received
0927 15:17:31.676863 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:31.676865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:31.676919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:31.677000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:31.677481 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.678017 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.678863 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.679133 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.679209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:31.681862 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.242.130:48641: BlockingRecv error: recv got EOF from 127.9.242.130:48641 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:31.768837 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 18938
I20260927 15:17:31.773974 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 19091
I20260927 15:17:31.778451 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 19223
I20260927 15:17:31.782613 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 18868
2026-09-27T15:17:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17853 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T15:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:31Z Disabled control of system clock
I20260927 15:17:31.809417 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37127
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 15:17:31.917313 19419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:31.917574 19419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:31.917606 19419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:31.921584 19419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:31.921666 19419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:31.921684 19419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:31.921698 19419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:31.921711 19419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:31.926467 19419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:37127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:31.927438 19419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:31.928714 19419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:31.936820 19427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:31.936848 19424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:31.936942 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:31.937003 19419 server_base.cc:1034] running on GCE node
I20260927 15:17:31.937644 19419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:31.938730 19419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:31.939889 19419 hybrid_clock.cc:648] HybridClock initialized: now 1790522251939866 us; error 37 us; skew 500 ppm
I20260927 15:17:31.941641 19419 webserver.cc:460] Webserver started at http://127.9.242.190:46297/ using document root <none> and password file <none>
I20260927 15:17:31.942212 19419 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:31.942276 19419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:31.942473 19419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:31.944439 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "d19abc3007054b0395c81816dff8c1ae"
format_stamp: "Formatted at 2026-09-27 15:17:31 on dist-test-slave-m4lt"
server_key: "eed74883b89b158547f68346d80fd673"
server_key_iv: "9a960a366acd4c80a04f5191b38445d9"
server_key_version: "encryptionkey@0"
I20260927 15:17:31.944960 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "d19abc3007054b0395c81816dff8c1ae"
format_stamp: "Formatted at 2026-09-27 15:17:31 on dist-test-slave-m4lt"
server_key: "eed74883b89b158547f68346d80fd673"
server_key_iv: "9a960a366acd4c80a04f5191b38445d9"
server_key_version: "encryptionkey@0"
I20260927 15:17:31.948428 19419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 15:17:31.950873 19433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:31.952128 19419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:31.952239 19419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "d19abc3007054b0395c81816dff8c1ae"
format_stamp: "Formatted at 2026-09-27 15:17:31 on dist-test-slave-m4lt"
server_key: "eed74883b89b158547f68346d80fd673"
server_key_iv: "9a960a366acd4c80a04f5191b38445d9"
server_key_version: "encryptionkey@0"
I20260927 15:17:31.952334 19419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:31.968837 19419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:31.969540 19419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:31.969734 19419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:31.976313 19419 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:37127
I20260927 15:17:31.976372 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:37127 every 8 connection(s)
I20260927 15:17:31.977252 19419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:31.980535 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:31.984786 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 19419
I20260927 15:17:31.984886 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:17:31.985208 10186 external_mini_cluster.cc:1428] Setting key c4fd62a992b13faf6ddca96cf225fc59
I20260927 15:17:31.986243 19486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae: Bootstrap starting.
I20260927 15:17:31.988529 19486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:31.989248 19486 log.cc:824] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:31.991250 19486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae: No bootstrap required, opened a new log
I20260927 15:17:31.994189 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:31.986615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.9.242.190:37127)
0927 15:17:31.987236 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:31.987252 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:31.987287 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:17:31.987929 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:31.987947 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:31.988331 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:31.988662 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:31.989071 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.990262 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.991128 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:31.991616 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:31.991756 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:31.992504 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:31.992509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:31.992521 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:31.992894 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:31.992951 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:31.992969 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:31.993048 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:31.993183 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 15:17:31.993423 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260927 15:17:31.994815 19486 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19abc3007054b0395c81816dff8c1ae" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37127 } }
I20260927 15:17:31.995085 19486 raft_consensus.cc:374] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:31.995142 19486 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19abc3007054b0395c81816dff8c1ae, State: Initialized, Role: FOLLOWER
I20260927 15:17:31.995532 19486 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19abc3007054b0395c81816dff8c1ae" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37127 } }
I20260927 15:17:31.995645 19486 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:31.995702 19486 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:31.995800 19486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:31.996795 19486 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19abc3007054b0395c81816dff8c1ae" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37127 } }
I20260927 15:17:31.997123 19486 leader_election.cc:302] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19abc3007054b0395c81816dff8c1ae; no voters:
I20260927 15:17:31.997406 19486 leader_election.cc:288] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:31.997503 19491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:31.997798 19491 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [term 1 LEADER]: Becoming Leader. State: Replica: d19abc3007054b0395c81816dff8c1ae, State: Running, Role: LEADER
I20260927 15:17:31.998133 19491 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19abc3007054b0395c81816dff8c1ae" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37127 } }
I20260927 15:17:31.998585 19486 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:31.999637 19492 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d19abc3007054b0395c81816dff8c1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19abc3007054b0395c81816dff8c1ae" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37127 } } }
I20260927 15:17:31.999612 19493 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19abc3007054b0395c81816dff8c1ae. Latest consensus state: current_term: 1 leader_uuid: "d19abc3007054b0395c81816dff8c1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19abc3007054b0395c81816dff8c1ae" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37127 } } }
I20260927 15:17:31.999750 19492 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:31.999750 19493 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:32.000090 19500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:32.003373 19500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:32.008792 19500 catalog_manager.cc:1347] Generated new cluster ID: 3eeb802dd9a74d9eb0d2d102f4c04032
I20260927 15:17:32.008876 19500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:32.024950 19500 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:32.026268 19500 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:32.034876 19500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d19abc3007054b0395c81816dff8c1ae: Generated new TSK 0
I20260927 15:17:32.035730 19500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:17:32.040717 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37127
--builtin_ntp_servers=127.9.242.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 15:17:32.174261 19510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:32.174525 19510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:32.174562 19510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:32.178661 19510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:32.178714 19510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:32.178767 19510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:32.183445 19510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:32.184463 19510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:32.185853 19510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:32.195297 19516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.195305 19515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:32.195456 19510 server_base.cc:1034] running on GCE node
W20260927 15:17:32.195305 19518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:32.195935 19510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:32.196640 19510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:32.197788 19510 hybrid_clock.cc:648] HybridClock initialized: now 1790522252197765 us; error 33 us; skew 500 ppm
I20260927 15:17:32.199718 19510 webserver.cc:460] Webserver started at http://127.9.242.129:41183/ using document root <none> and password file <none>
I20260927 15:17:32.200323 19510 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:32.200383 19510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:32.200582 19510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:32.202692 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "a1ee20e45e2446cba21c5cdaab3f7995"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "4b56ca339c4d8a680ec7ff549d4a040e"
server_key_iv: "f23698cbe19541a799607b55e2be3ad2"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.203227 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "a1ee20e45e2446cba21c5cdaab3f7995"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "4b56ca339c4d8a680ec7ff549d4a040e"
server_key_iv: "f23698cbe19541a799607b55e2be3ad2"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.206808 19510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:17:32.209288 19524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.210573 19510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 15:17:32.210673 19510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "a1ee20e45e2446cba21c5cdaab3f7995"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "4b56ca339c4d8a680ec7ff549d4a040e"
server_key_iv: "f23698cbe19541a799607b55e2be3ad2"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.210777 19510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:32.223467 19510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:32.224215 19510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:32.224364 19510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:32.224913 19510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:32.225889 19510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:32.225945 19510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.225993 19510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:32.226013 19510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.234330 19510 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:42411
I20260927 15:17:32.234392 19637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:42411 every 8 connection(s)
I20260927 15:17:32.235258 19510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:32.237325 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 19510
I20260927 15:17:32.237433 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:32.237730 10186 external_mini_cluster.cc:1428] Setting key 617ce019b667a04224edd57eb7602e24
I20260927 15:17:32.239813 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37127
--builtin_ntp_servers=127.9.242.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:17:32.245975 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:32.237182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:38717 (local address 127.9.242.190:37127)
0927 15:17:32.237340 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:32.237345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:32.238544 (+ 1199us) server_negotiation.cc:407] Connection header received
0927 15:17:32.240216 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:32.240218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:32.240262 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:32.240319 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:32.241465 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.242290 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.243933 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.244168 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.244249 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:32.245507 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:32.245510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:32.245513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:32.245525 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:32.245549 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:32.245557 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:32.245612 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:32.245803 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 15:17:32.245855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:17:32.247216 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:32.237613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37127 (local address 127.9.242.129:38717)
0927 15:17:32.238376 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:32.238435 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:32.239965 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:32.240508 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:32.240518 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:32.240629 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:32.241321 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:32.241338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.242450 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.242454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:32.243800 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:32.243813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.244327 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.244332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:32.244551 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:32.245172 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:32.245174 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:32.245233 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:32.245342 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:32.245633 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:32.245642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:32.245648 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:17:32.245808 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 15:17:32.246135 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260927 15:17:32.248389 19638 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37127
I20260927 15:17:32.248646 19638 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:32.249148 19638 heartbeater.cc:511] Master 127.9.242.190:37127 requested a full tablet report, sending...
I20260927 15:17:32.250559 19450 ts_manager.cc:194] Registered new tserver with Master: a1ee20e45e2446cba21c5cdaab3f7995 (127.9.242.129:42411)
I20260927 15:17:32.251669 19450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.129:38717
W20260927 15:17:32.350224 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:32.350495 19641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:32.350535 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:32.354588 19641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:32.354655 19641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:32.354714 19641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:32.359611 19641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:32.360708 19641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:32.362190 19641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:32.371721 19646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.371762 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.371750 19647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:32.372081 19641 server_base.cc:1034] running on GCE node
I20260927 15:17:32.372498 19641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:32.373169 19641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:32.374361 19641 hybrid_clock.cc:648] HybridClock initialized: now 1790522252374345 us; error 33 us; skew 500 ppm
I20260927 15:17:32.376390 19641 webserver.cc:460] Webserver started at http://127.9.242.130:36947/ using document root <none> and password file <none>
I20260927 15:17:32.376952 19641 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:32.377023 19641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:32.377223 19641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:32.379266 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "7ee29a33671d40519b51823d9d685404"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "a26090449284754168e0dbe1d9d79359"
server_key_iv: "a0b7b4b7fcac80fe2919233edcab9f24"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.379787 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "7ee29a33671d40519b51823d9d685404"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "a26090449284754168e0dbe1d9d79359"
server_key_iv: "a0b7b4b7fcac80fe2919233edcab9f24"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.383460 19641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:17:32.385995 19655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.387187 19641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:32.387290 19641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "7ee29a33671d40519b51823d9d685404"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "a26090449284754168e0dbe1d9d79359"
server_key_iv: "a0b7b4b7fcac80fe2919233edcab9f24"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.387390 19641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:32.401378 19641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:32.402061 19641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:32.402223 19641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:32.402772 19641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:32.403685 19641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:32.403730 19641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.403780 19641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:32.403808 19641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.411826 19641 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:42115
I20260927 15:17:32.411856 19768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:42115 every 8 connection(s)
I20260927 15:17:32.412705 19641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:32.416442 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 19641
I20260927 15:17:32.416582 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:32.416836 10186 external_mini_cluster.cc:1428] Setting key 884aba6eb8ae5f6b42caf1cbf3fdb973
I20260927 15:17:32.418773 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37127
--builtin_ntp_servers=127.9.242.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 15:17:32.422302 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:32.414600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:34153 (local address 127.9.242.190:37127)
0927 15:17:32.414738 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:32.414743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:32.415856 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 15:17:32.417352 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:32.417356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:32.417404 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:32.417483 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:32.418605 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.419384 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.420496 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.420705 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.420792 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:32.421850 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:32.421854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:32.421856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:32.421867 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:32.421886 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:32.421893 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:32.421950 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:32.422102 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 15:17:32.422161 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:17:32.423242 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:32.415053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37127 (local address 127.9.242.130:34153)
0927 15:17:32.415710 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:32.415751 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:32.417109 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:32.417687 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:32.417696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:32.417800 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:32.418472 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:32.418486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.419500 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.419503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:32.420364 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:32.420372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.420834 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.420838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:32.420996 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:32.421569 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:32.421571 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:32.421610 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:32.421707 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:32.421962 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:32.421970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:32.421976 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:17:32.422089 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 15:17:32.422370 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260927 15:17:32.424590 19769 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37127
I20260927 15:17:32.424803 19769 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:32.425287 19769 heartbeater.cc:511] Master 127.9.242.190:37127 requested a full tablet report, sending...
I20260927 15:17:32.426362 19450 ts_manager.cc:194] Registered new tserver with Master: 7ee29a33671d40519b51823d9d685404 (127.9.242.130:42115)
I20260927 15:17:32.427117 19450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.130:34153
W20260927 15:17:32.528086 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:32.528342 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:32.528378 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:32.532346 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:32.532407 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:32.532461 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:32.537374 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:32.538434 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:32.539739 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:32.549208 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.549196 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.549233 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:32.549247 19772 server_base.cc:1034] running on GCE node
I20260927 15:17:32.549824 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:32.550441 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:32.551635 19772 hybrid_clock.cc:648] HybridClock initialized: now 1790522252551615 us; error 32 us; skew 500 ppm
I20260927 15:17:32.553448 19772 webserver.cc:460] Webserver started at http://127.9.242.131:42781/ using document root <none> and password file <none>
I20260927 15:17:32.554067 19772 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:32.554134 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:32.554327 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:32.556465 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "3f8f2ba3cbdb465eb5867f0f7c28222a"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "d2e25f5c80fd0b17cfc467c822c76937"
server_key_iv: "86d032b71e5ba4acf9bea32f7daa7d7d"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.557020 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "3f8f2ba3cbdb465eb5867f0f7c28222a"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "d2e25f5c80fd0b17cfc467c822c76937"
server_key_iv: "86d032b71e5ba4acf9bea32f7daa7d7d"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.560603 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:17:32.563198 19786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.564457 19772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:17:32.564584 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "3f8f2ba3cbdb465eb5867f0f7c28222a"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "d2e25f5c80fd0b17cfc467c822c76937"
server_key_iv: "86d032b71e5ba4acf9bea32f7daa7d7d"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.564677 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:32.585704 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:32.586410 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:32.586580 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:32.587172 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:32.588104 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:32.588150 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.588202 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:32.588231 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.596403 19772 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:41517
I20260927 15:17:32.596478 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:41517 every 8 connection(s)
I20260927 15:17:32.597278 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:32.604766 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 19772
I20260927 15:17:32.604872 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:32.605189 10186 external_mini_cluster.cc:1428] Setting key f8c87576aad7213de5ee4de208ed431d
I20260927 15:17:32.606567 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:32.599040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:35959 (local address 127.9.242.190:37127)
0927 15:17:32.599247 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:32.599252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:32.600430 (+ 1178us) server_negotiation.cc:407] Connection header received
0927 15:17:32.601818 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:32.601821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:32.601867 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:32.601931 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:32.602770 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.603338 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.604534 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.604757 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.604858 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:32.606127 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:32.606131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:32.606134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:32.606148 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:32.606168 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:32.606176 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:32.606231 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:32.606400 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 15:17:32.606455 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 15:17:32.607528 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:32.599504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37127 (local address 127.9.242.131:35959)
0927 15:17:32.600296 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:32.600337 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:32.601576 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:32.602074 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:32.602080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:32.602162 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:32.602632 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:32.602644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.603487 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.603490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:32.604409 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:32.604419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:32.604952 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:32.604957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:32.605159 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:32.605856 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:32.605858 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:32.605925 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:32.605993 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:32.606248 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:32.606255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:32.606261 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:17:32.606387 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 15:17:32.606666 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":112,"threads_started":1}
I20260927 15:17:32.608501 19900 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37127
I20260927 15:17:32.608712 19900 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:32.609181 19900 heartbeater.cc:511] Master 127.9.242.190:37127 requested a full tablet report, sending...
I20260927 15:17:32.610205 19450 ts_manager.cc:194] Registered new tserver with Master: 3f8f2ba3cbdb465eb5867f0f7c28222a (127.9.242.131:41517)
I20260927 15:17:32.610817 19450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.131:35959
I20260927 15:17:32.618830 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:17:32.621476 19489 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:32.620625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.9.242.190:37127)
0927 15:17:32.620777 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:32.620781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:32.620922 (+ 141us) server_negotiation.cc:407] Connection header received
0927 15:17:32.621014 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:32.621016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:32.621066 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:32.621139 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:32.621348 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44800: BlockingRecv error: recv got EOF from 127.0.0.1:44800 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 15:17:32.621810 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 19510
I20260927 15:17:32.626338 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 19641
I20260927 15:17:32.631036 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 19772
I20260927 15:17:32.635566 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 19419
2026-09-27T15:17:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (852 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T15:17:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:32Z Disabled control of system clock
I20260927 15:17:32.662180 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43447
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:43075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:43447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:17:32.767779 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:32.768090 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:32.768126 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:32.772034 19911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:32.772086 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:32.772101 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:32.772116 19911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:32.772130 19911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:32.776859 19911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:43447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:32.777901 19911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:32.779232 19911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:32.787096 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:32.787084 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:32.787225 19911 server_base.cc:1034] running on GCE node
W20260927 15:17:32.787101 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:32.787875 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:32.788898 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:32.790096 19911 hybrid_clock.cc:648] HybridClock initialized: now 1790522252790080 us; error 29 us; skew 500 ppm
I20260927 15:17:32.791826 19911 webserver.cc:460] Webserver started at http://127.9.242.190:32991/ using document root <none> and password file <none>
I20260927 15:17:32.792374 19911 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:32.792439 19911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:32.792621 19911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:32.794667 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "d9b7601e8541459f91c433b64aecfeab"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "3e9497d54f612fe9b2676106720eb94d"
server_key_iv: "99d48082fc1ecb602efb51bbfd68e881"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.795205 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "d9b7601e8541459f91c433b64aecfeab"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "3e9497d54f612fe9b2676106720eb94d"
server_key_iv: "99d48082fc1ecb602efb51bbfd68e881"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.798797 19911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 15:17:32.801131 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:32.802455 19911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:17:32.802568 19911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "d9b7601e8541459f91c433b64aecfeab"
format_stamp: "Formatted at 2026-09-27 15:17:32 on dist-test-slave-m4lt"
server_key: "3e9497d54f612fe9b2676106720eb94d"
server_key_iv: "99d48082fc1ecb602efb51bbfd68e881"
server_key_version: "encryptionkey@0"
I20260927 15:17:32.802663 19911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:32.806970 19911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:32.807518 19911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:32.807662 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:32.813627 19911 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:43447
I20260927 15:17:32.813633 19977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:43447 every 8 connection(s)
I20260927 15:17:32.814493 19911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:32.817227 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 19911
I20260927 15:17:32.817317 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:17:32.817480 19978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:32.817585 10186 external_mini_cluster.cc:1428] Setting key 14bebdff654b05c3984d4b2c58249367
I20260927 15:17:32.822956 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:32.818946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.9.242.190:43447)
0927 15:17:32.819514 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:32.819530 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:32.819565 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:17:32.820331 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:32.820347 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:32.820681 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:32.820947 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:32.821732 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:32.821736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:32.821748 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:32.821758 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:32.821813 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:32.821829 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:32.821932 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:32.822076 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 15:17:32.822308 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260927 15:17:32.823626 19978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab: Bootstrap starting.
I20260927 15:17:32.826025 19978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:32.826699 19978 log.cc:824] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:32.828558 19978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab: No bootstrap required, opened a new log
I20260927 15:17:32.832314 19978 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b7601e8541459f91c433b64aecfeab" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43447 } }
I20260927 15:17:32.832537 19978 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:32.832600 19978 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b7601e8541459f91c433b64aecfeab, State: Initialized, Role: FOLLOWER
I20260927 15:17:32.833003 19978 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b7601e8541459f91c433b64aecfeab" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43447 } }
I20260927 15:17:32.833128 19978 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:32.833189 19978 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:32.833280 19978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:32.834221 19978 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b7601e8541459f91c433b64aecfeab" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43447 } }
I20260927 15:17:32.834542 19978 leader_election.cc:302] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9b7601e8541459f91c433b64aecfeab; no voters:
I20260927 15:17:32.834836 19978 leader_election.cc:288] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:32.834954 19983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:32.835171 19983 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [term 1 LEADER]: Becoming Leader. State: Replica: d9b7601e8541459f91c433b64aecfeab, State: Running, Role: LEADER
I20260927 15:17:32.835511 19983 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b7601e8541459f91c433b64aecfeab" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43447 } }
I20260927 15:17:32.835990 19978 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:32.836989 19984 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b7601e8541459f91c433b64aecfeab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b7601e8541459f91c433b64aecfeab" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43447 } } }
I20260927 15:17:32.837109 19984 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:32.837495 19985 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b7601e8541459f91c433b64aecfeab. Latest consensus state: current_term: 1 leader_uuid: "d9b7601e8541459f91c433b64aecfeab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b7601e8541459f91c433b64aecfeab" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43447 } } }
I20260927 15:17:32.837677 19985 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:32.838162 19992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:32.840741 19992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:32.846364 19992 catalog_manager.cc:1347] Generated new cluster ID: 67f280bd41f54d9f8834644e554f86d1
I20260927 15:17:32.846436 19992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:32.861224 19992 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:32.861564 19992 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:32.870009 19992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9b7601e8541459f91c433b64aecfeab: Generated new TSK 0
I20260927 15:17:32.870765 19992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:17:32.880354 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43447
--builtin_ntp_servers=127.9.242.148:43075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 15:17:32.990796 20002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:32.991050 20002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:32.991089 20002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:32.995095 20002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:32.995149 20002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:32.995215 20002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:32.999926 20002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:33.001030 20002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:33.002473 20002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:33.012025 20010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.012059 20007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:33.012130 20002 server_base.cc:1034] running on GCE node
W20260927 15:17:33.012019 20008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:33.012660 20002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:33.013337 20002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:33.014504 20002 hybrid_clock.cc:648] HybridClock initialized: now 1790522253014472 us; error 47 us; skew 500 ppm
I20260927 15:17:33.016428 20002 webserver.cc:460] Webserver started at http://127.9.242.129:38817/ using document root <none> and password file <none>
I20260927 15:17:33.017002 20002 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:33.017061 20002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:33.017258 20002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:33.019389 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "e9109753beba4a11a691ef4fa8408660"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "cfe037bf9d8ddd8e3961a757c7fe7859"
server_key_iv: "9491467b6da214377d83f7c593576e09"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.019945 20002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "e9109753beba4a11a691ef4fa8408660"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "cfe037bf9d8ddd8e3961a757c7fe7859"
server_key_iv: "9491467b6da214377d83f7c593576e09"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.023617 20002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:17:33.026309 20016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.027627 20002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:33.027735 20002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "e9109753beba4a11a691ef4fa8408660"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "cfe037bf9d8ddd8e3961a757c7fe7859"
server_key_iv: "9491467b6da214377d83f7c593576e09"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.027839 20002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:33.032266 20002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:33.032822 20002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:33.032974 20002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:33.033471 20002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:33.034353 20002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:33.034396 20002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.034431 20002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:33.034446 20002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.042748 20002 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:45907
I20260927 15:17:33.042773 20129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:45907 every 8 connection(s)
I20260927 15:17:33.043646 20002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:33.045984 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 20002
I20260927 15:17:33.046090 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:17:33.046361 10186 external_mini_cluster.cc:1428] Setting key e5ca1d95b7a7f7a4134b8d7dedd45273
I20260927 15:17:33.048333 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43447
--builtin_ntp_servers=127.9.242.148:43075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:17:33.049795 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.045391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:36437 (local address 127.9.242.190:43447)
0927 15:17:33.045512 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:33.045516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:33.046683 (+ 1167us) server_negotiation.cc:407] Connection header received
0927 15:17:33.048155 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:33.048158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:33.048197 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:33.048284 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:33.049363 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:33.049367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:33.049370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:33.049384 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:33.049409 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:33.049419 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:33.049507 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:33.049633 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 15:17:33.049693 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:33.050827 20132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.045825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43447 (local address 127.9.242.129:36437)
0927 15:17:33.046519 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.046568 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:33.047946 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:33.048416 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:33.048422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:33.048536 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:33.049118 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:33.049119 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:33.049157 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:33.049224 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:33.049504 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:33.049511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:33.049516 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:17:33.049696 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 15:17:33.049935 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
I20260927 15:17:33.051896 20130 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43447
I20260927 15:17:33.052145 20130 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:33.052590 20130 heartbeater.cc:511] Master 127.9.242.190:43447 requested a full tablet report, sending...
I20260927 15:17:33.054208 19942 ts_manager.cc:194] Registered new tserver with Master: e9109753beba4a11a691ef4fa8408660 (127.9.242.129:45907)
W20260927 15:17:33.158181 20133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:33.158439 20133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:33.158493 20133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:33.162462 20133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:33.162518 20133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:33.162593 20133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:33.167397 20133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:33.168473 20133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:33.169920 20133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:33.178969 20139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.179000 20138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.179028 20141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:33.179247 20133 server_base.cc:1034] running on GCE node
I20260927 15:17:33.179628 20133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:33.180197 20133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:33.181363 20133 hybrid_clock.cc:648] HybridClock initialized: now 1790522253181342 us; error 37 us; skew 500 ppm
I20260927 15:17:33.183056 20133 webserver.cc:460] Webserver started at http://127.9.242.130:37171/ using document root <none> and password file <none>
I20260927 15:17:33.183585 20133 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:33.183647 20133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:33.183828 20133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:33.185858 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "3315dac5ee8d4c84a72b283ec543f0d7"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "e5f7a5608b72eeb42f7c5c4bdcca2c76"
server_key_iv: "af64acf970d787e696351e4b1c399d6b"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.186385 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "3315dac5ee8d4c84a72b283ec543f0d7"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "e5f7a5608b72eeb42f7c5c4bdcca2c76"
server_key_iv: "af64acf970d787e696351e4b1c399d6b"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.189968 20133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:17:33.192396 20147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.193683 20133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:33.193790 20133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "3315dac5ee8d4c84a72b283ec543f0d7"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "e5f7a5608b72eeb42f7c5c4bdcca2c76"
server_key_iv: "af64acf970d787e696351e4b1c399d6b"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.193884 20133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:33.198171 20133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:33.198734 20133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:33.198879 20133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:33.199370 20133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:33.200210 20133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:33.200254 20133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.200304 20133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:33.200331 20133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.208501 20133 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:45799
I20260927 15:17:33.208524 20260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:45799 every 8 connection(s)
I20260927 15:17:33.209396 20133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:33.214526 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 20133
I20260927 15:17:33.214650 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:33.214910 10186 external_mini_cluster.cc:1428] Setting key cfdd8f4aa158c49e05567661f6e0065c
I20260927 15:17:33.215415 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.211186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:48491 (local address 127.9.242.190:43447)
0927 15:17:33.211330 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:33.211333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:33.212423 (+ 1090us) server_negotiation.cc:407] Connection header received
0927 15:17:33.213714 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:33.213718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:33.213759 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:33.213832 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:33.214968 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:33.214971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:33.214973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:33.214985 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:33.215007 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:33.215016 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:33.215106 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:33.215241 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 15:17:33.215282 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:17:33.216722 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.211537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43447 (local address 127.9.242.130:48491)
0927 15:17:33.212269 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.212310 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:33.213494 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:33.213994 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:33.214000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:33.214095 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:33.214710 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:33.214713 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:33.214765 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:33.214834 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:33.215109 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:33.215118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:33.215124 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:17:33.215287 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 15:17:33.215527 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260927 15:17:33.216738 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43447
--builtin_ntp_servers=127.9.242.148:43075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 15:17:33.217800 20261 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43447
I20260927 15:17:33.218019 20261 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:33.218447 20261 heartbeater.cc:511] Master 127.9.242.190:43447 requested a full tablet report, sending...
I20260927 15:17:33.219444 19942 ts_manager.cc:194] Registered new tserver with Master: 3315dac5ee8d4c84a72b283ec543f0d7 (127.9.242.130:45799)
W20260927 15:17:33.323184 20264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:33.323449 20264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:33.323503 20264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:33.327553 20264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:33.327618 20264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:33.327696 20264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:33.332579 20264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:33.333616 20264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:33.334957 20264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:33.344348 20270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.344336 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:33.344341 20272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:33.344417 20264 server_base.cc:1034] running on GCE node
I20260927 15:17:33.345010 20264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:33.345646 20264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:33.346843 20264 hybrid_clock.cc:648] HybridClock initialized: now 1790522253346821 us; error 33 us; skew 500 ppm
I20260927 15:17:33.348596 20264 webserver.cc:460] Webserver started at http://127.9.242.131:34903/ using document root <none> and password file <none>
I20260927 15:17:33.349136 20264 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:33.349198 20264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:33.349375 20264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:33.351393 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "adb06e78a2514b3ab862fd91aee63985"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "9f14f55d9650382e87405a6b4ec2c28f"
server_key_iv: "facd614fd9467ae251bb2acf5247bd54"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.351914 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "adb06e78a2514b3ab862fd91aee63985"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "9f14f55d9650382e87405a6b4ec2c28f"
server_key_iv: "facd614fd9467ae251bb2acf5247bd54"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.355497 20264 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:17:33.357991 20278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.359222 20264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:17:33.359334 20264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "adb06e78a2514b3ab862fd91aee63985"
format_stamp: "Formatted at 2026-09-27 15:17:33 on dist-test-slave-m4lt"
server_key: "9f14f55d9650382e87405a6b4ec2c28f"
server_key_iv: "facd614fd9467ae251bb2acf5247bd54"
server_key_version: "encryptionkey@0"
I20260927 15:17:33.359431 20264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:33.363876 20264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:33.364456 20264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:33.364610 20264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:33.365103 20264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:33.366003 20264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:33.366052 20264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.366101 20264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:33.366122 20264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:33.374487 20264 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:46821
I20260927 15:17:33.374567 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:46821 every 8 connection(s)
I20260927 15:17:33.375373 20264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:33.381100 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.377051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:49803 (local address 127.9.242.190:43447)
0927 15:17:33.377166 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:33.377170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:33.378272 (+ 1102us) server_negotiation.cc:407] Connection header received
0927 15:17:33.379522 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:33.379527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:33.379572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:33.379651 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:33.380680 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:33.380684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:33.380687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:17:33.380700 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:33.380719 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:17:33.380728 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:33.380802 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:33.380916 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 15:17:33.380959 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:17:33.381973 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:33.377401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43447 (local address 127.9.242.131:49803)
0927 15:17:33.378127 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:33.378168 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:33.379321 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:33.379819 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:33.379826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:33.379915 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:33.380424 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:17:33.380426 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:33.380465 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:17:33.380539 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:33.380792 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:33.380799 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:33.380804 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:17:33.380954 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 15:17:33.381188 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":114,"threads_started":1}
I20260927 15:17:33.382833 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 20264
I20260927 15:17:33.382918 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:33.383072 20392 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43447
I20260927 15:17:33.383168 10186 external_mini_cluster.cc:1428] Setting key b53edf77bc7a1204ad6a704164e8e8a5
I20260927 15:17:33.383284 20392 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:33.383658 20392 heartbeater.cc:511] Master 127.9.242.190:43447 requested a full tablet report, sending...
I20260927 15:17:33.384601 19941 ts_manager.cc:194] Registered new tserver with Master: adb06e78a2514b3ab862fd91aee63985 (127.9.242.131:46821)
I20260927 15:17:33.396715 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 15:17:33.399621 19981 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:17:33.398728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.9.242.190:43447)
0927 15:17:33.398858 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:33.398863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:33.399118 (+ 255us) server_negotiation.cc:407] Connection header received
0927 15:17:33.399162 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:33.399164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:33.399199 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:33.399269 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:33.399494 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33572: BlockingRecv error: recv got EOF from 127.0.0.1:33572 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:17:33.399760 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 20002
I20260927 15:17:33.405233 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 20133
I20260927 15:17:33.409592 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 20264
I20260927 15:17:33.413691 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 19911
2026-09-27T15:17:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (777 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:33 dist-test-slave-m4lt krb5kdc[20402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:33 dist-test-slave-m4lt krb5kdc[20402](info): set up 2 sockets
Sep 27 15:17:33 dist-test-slave-m4lt krb5kdc[20402](info): commencing operation
krb5kdc: starting...
W20260927 15:17:34.362315 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:17:34 dist-test-slave-m4lt krb5kdc[20402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:17:34.480083 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36557
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:36557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:34.586019 20415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:34.586304 20415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:34.586346 20415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:34.590399 20415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:17:34.590457 20415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:34.590476 20415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:34.590490 20415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:17:34.590507 20415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:17:34.595821 20415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:36557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:34.596858 20415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:34.597898 20415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:34.606349 20420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:34.606349 20421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:34.606420 20423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:34.606746 20415 server_base.cc:1034] running on GCE node
I20260927 15:17:34.607260 20415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:34.608346 20415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:34.609494 20415 hybrid_clock.cc:648] HybridClock initialized: now 1790522254609479 us; error 31 us; skew 500 ppm
Sep 27 15:17:34 dist-test-slave-m4lt krb5kdc[20402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:34.612114 20415 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:17:34.612974 20415 webserver.cc:460] Webserver started at http://127.9.242.190:37843/ using document root <none> and password file <none>
I20260927 15:17:34.613488 20415 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:34.613559 20415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:34.613785 20415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:34.615777 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "6fc1b4172e614a16904879f25e9ab375"
format_stamp: "Formatted at 2026-09-27 15:17:34 on dist-test-slave-m4lt"
server_key: "9e8ad57c976a69206723c788b3bff5ca"
server_key_iv: "655e73e6e237f45e64c50e0cf8383e7d"
server_key_version: "encryptionkey@0"
I20260927 15:17:34.616302 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "6fc1b4172e614a16904879f25e9ab375"
format_stamp: "Formatted at 2026-09-27 15:17:34 on dist-test-slave-m4lt"
server_key: "9e8ad57c976a69206723c788b3bff5ca"
server_key_iv: "655e73e6e237f45e64c50e0cf8383e7d"
server_key_version: "encryptionkey@0"
I20260927 15:17:34.619903 20415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:17:34.622290 20430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:34.623596 20415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:34.623709 20415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "6fc1b4172e614a16904879f25e9ab375"
format_stamp: "Formatted at 2026-09-27 15:17:34 on dist-test-slave-m4lt"
server_key: "9e8ad57c976a69206723c788b3bff5ca"
server_key_iv: "655e73e6e237f45e64c50e0cf8383e7d"
server_key_version: "encryptionkey@0"
I20260927 15:17:34.623806 20415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:34.637549 20415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:34.640723 20415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:34.640894 20415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:34.648078 20415 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:36557
I20260927 15:17:34.648357 20482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:36557 every 8 connection(s)
I20260927 15:17:34.649096 20415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:17:34.651789 20483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:34.656023 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 20415
I20260927 15:17:34.656141 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:17:34.656400 10186 external_mini_cluster.cc:1428] Setting key b4a0ff56bd40430a4d09eda29995dfe0
I20260927 15:17:34.657335 20483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375: Bootstrap starting.
I20260927 15:17:34.659601 20483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:34.660282 20483 log.cc:824] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:34.662078 20483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375: No bootstrap required, opened a new log
Sep 27 15:17:34 dist-test-slave-m4lt krb5kdc[20402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:34.665882 20483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1b4172e614a16904879f25e9ab375" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36557 } }
I20260927 15:17:34.666129 20483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:34.666209 20483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc1b4172e614a16904879f25e9ab375, State: Initialized, Role: FOLLOWER
I20260927 15:17:34.666700 20483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1b4172e614a16904879f25e9ab375" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36557 } }
I20260927 15:17:34.666857 20483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:17:34.666923 20483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:17:34.667007 20483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:34.667996 20483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1b4172e614a16904879f25e9ab375" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36557 } }
I20260927 15:17:34.668346 20483 leader_election.cc:302] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fc1b4172e614a16904879f25e9ab375; no voters:
I20260927 15:17:34.668645 20483 leader_election.cc:288] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:17:34.668787 20488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:34.669061 20488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [term 1 LEADER]: Becoming Leader. State: Replica: 6fc1b4172e614a16904879f25e9ab375, State: Running, Role: LEADER
I20260927 15:17:34.669452 20488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1b4172e614a16904879f25e9ab375" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36557 } }
I20260927 15:17:34.669881 20483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:17:34.670784 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.657866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.9.242.190:36557)
0927 15:17:34.658280 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:34.658294 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:34.658328 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:17:34.659023 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:34.659042 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:34.659367 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:34.659679 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:34.660421 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:34.661393 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:34.662268 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:34.662958 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:34.663123 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:34.665581 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:34.665587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:34.665603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:34.665633 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:34.667640 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:34.668062 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:34.668067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:34.668073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:34.668204 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:34.668470 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:34.668473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:34.668474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:34.668977 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:34.669149 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:34.669490 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 15:17:34.669820 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260927 15:17:34.671084 20490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fc1b4172e614a16904879f25e9ab375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1b4172e614a16904879f25e9ab375" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36557 } } }
I20260927 15:17:34.671204 20490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:34.671543 20489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fc1b4172e614a16904879f25e9ab375. Latest consensus state: current_term: 1 leader_uuid: "6fc1b4172e614a16904879f25e9ab375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fc1b4172e614a16904879f25e9ab375" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36557 } } }
I20260927 15:17:34.671617 20489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375 [sys.catalog]: This master's current role is: LEADER
I20260927 15:17:34.671989 20499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:17:34.675251 20499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:17:34.680763 20499 catalog_manager.cc:1347] Generated new cluster ID: 3e4e96c5d7134858b6f072c41608f85a
I20260927 15:17:34.680832 20499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:17:34.703243 20499 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:17:34.704185 20499 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:17:34.709311 20499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375: Generated new TSK 0
I20260927 15:17:34.709980 20499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:17:34.735967 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36557
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:34.845052 20509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:34.845320 20509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:34.845362 20509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:34.849354 20509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:34.849424 20509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:34.849483 20509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:17:34.855098 20509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:34.856153 20509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:34.857232 20509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:34.866889 20515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:34.866883 20514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:34.866883 20517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:34.867084 20509 server_base.cc:1034] running on GCE node
I20260927 15:17:34.867540 20509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:34.868189 20509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:34.869359 20509 hybrid_clock.cc:648] HybridClock initialized: now 1790522254869339 us; error 31 us; skew 500 ppm
Sep 27 15:17:34 dist-test-slave-m4lt krb5kdc[20402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:34.872249 20509 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:17:34.873109 20509 webserver.cc:460] Webserver started at http://127.9.242.129:33187/ using document root <none> and password file <none>
I20260927 15:17:34.873698 20509 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:34.873766 20509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:34.873934 20509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:34.875916 20509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "bee5c0e4c84a49339e427fe6d14747ef"
format_stamp: "Formatted at 2026-09-27 15:17:34 on dist-test-slave-m4lt"
server_key: "4c251809275e496007a5ab0e877266d3"
server_key_iv: "e09c3efcedfda1fb6dc222ee6918cf12"
server_key_version: "encryptionkey@0"
I20260927 15:17:34.876417 20509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "bee5c0e4c84a49339e427fe6d14747ef"
format_stamp: "Formatted at 2026-09-27 15:17:34 on dist-test-slave-m4lt"
server_key: "4c251809275e496007a5ab0e877266d3"
server_key_iv: "e09c3efcedfda1fb6dc222ee6918cf12"
server_key_version: "encryptionkey@0"
I20260927 15:17:34.880065 20509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:17:34.882527 20524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:34.883700 20509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:34.883796 20509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "bee5c0e4c84a49339e427fe6d14747ef"
format_stamp: "Formatted at 2026-09-27 15:17:34 on dist-test-slave-m4lt"
server_key: "4c251809275e496007a5ab0e877266d3"
server_key_iv: "e09c3efcedfda1fb6dc222ee6918cf12"
server_key_version: "encryptionkey@0"
I20260927 15:17:34.883911 20509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:34.898164 20509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:34.900924 20509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:34.901081 20509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:34.901635 20509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:34.902424 20509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:34.902467 20509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:34.902508 20509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:34.902540 20509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:34.910954 20509 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:42583
I20260927 15:17:34.910977 20637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:42583 every 8 connection(s)
I20260927 15:17:34.911826 20509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:17:34.921860 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 20509
I20260927 15:17:34.921968 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
Sep 27 15:17:34 dist-test-slave-m4lt krb5kdc[20402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:34.922271 10186 external_mini_cluster.cc:1428] Setting key 660f32230d74634a2d8f8124ad584cf9
I20260927 15:17:34.927073 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.913522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:57919 (local address 127.9.242.190:36557)
0927 15:17:34.913713 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:34.913718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:34.914558 (+ 840us) server_negotiation.cc:407] Connection header received
0927 15:17:34.915822 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:34.915825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:34.915866 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:34.915936 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:34.917083 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:34.917634 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:34.919331 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:34.919684 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:34.919785 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:34.922732 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:34.922736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:34.922739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:34.922768 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:34.924882 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:34.925383 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:34.925387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:34.925388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:34.925480 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:34.925886 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:34.925889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:34.925890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:34.926078 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:34.926163 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:34.926747 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 15:17:34.926907 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:17:34.927925 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:34.913978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36557 (local address 127.9.242.129:57919)
0927 15:17:34.914412 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:34.914453 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:34.915636 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:34.916121 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:34.916128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:34.916466 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:34.916918 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:34.916932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:34.917818 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:34.917823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:34.919209 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:34.919221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:34.919838 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:34.919842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:34.920040 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:34.920682 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:34.920702 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:34.922543 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:34.924990 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:34.924996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:34.925012 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:34.925267 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:34.925567 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:34.925570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:34.925572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:34.925787 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:34.926177 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:34.926185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:34.926492 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 15:17:34.926680 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 15:17:34.927002 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260927 15:17:34.929363 20638 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36557
I20260927 15:17:34.929610 20638 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:34.930156 20638 heartbeater.cc:511] Master 127.9.242.190:36557 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:17:34.931835 20443 ts_manager.cc:194] Registered new tserver with Master: bee5c0e4c84a49339e427fe6d14747ef (127.9.242.129:42583)
I20260927 15:17:34.933097 20443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:57919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:17:34.948251 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36557
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:35.057605 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:35.057920 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:35.057981 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:35.061985 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:35.062050 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:35.062126 20643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:17:35.067377 20643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:35.068392 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:35.069377 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:35.078665 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:35.078682 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:35.078723 20643 server_base.cc:1034] running on GCE node
W20260927 15:17:35.078686 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:35.079370 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:35.079936 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:35.081112 20643 hybrid_clock.cc:648] HybridClock initialized: now 1790522255081094 us; error 31 us; skew 500 ppm
Sep 27 15:17:35 dist-test-slave-m4lt krb5kdc[20402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522255, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:35.083719 20643 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:17:35.084504 20643 webserver.cc:460] Webserver started at http://127.9.242.130:36565/ using document root <none> and password file <none>
I20260927 15:17:35.084992 20643 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:35.085053 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:35.085225 20643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:35.087204 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "858199e80a6e4a18b968d555e1a72c8e"
format_stamp: "Formatted at 2026-09-27 15:17:35 on dist-test-slave-m4lt"
server_key: "89b9099acf1a15fb8b4ed61928f12cc0"
server_key_iv: "6017c5ef256aa326454e40caf5cbe7c1"
server_key_version: "encryptionkey@0"
I20260927 15:17:35.087723 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "858199e80a6e4a18b968d555e1a72c8e"
format_stamp: "Formatted at 2026-09-27 15:17:35 on dist-test-slave-m4lt"
server_key: "89b9099acf1a15fb8b4ed61928f12cc0"
server_key_iv: "6017c5ef256aa326454e40caf5cbe7c1"
server_key_version: "encryptionkey@0"
I20260927 15:17:35.091207 20643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 15:17:35.093502 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:35.094784 20643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:17:35.094884 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "858199e80a6e4a18b968d555e1a72c8e"
format_stamp: "Formatted at 2026-09-27 15:17:35 on dist-test-slave-m4lt"
server_key: "89b9099acf1a15fb8b4ed61928f12cc0"
server_key_iv: "6017c5ef256aa326454e40caf5cbe7c1"
server_key_version: "encryptionkey@0"
I20260927 15:17:35.094970 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:35.116567 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:35.119251 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:35.119412 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:35.119987 20643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:35.120752 20643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:35.120797 20643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:35.120847 20643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:35.120874 20643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:35.129150 20643 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:43861
I20260927 15:17:35.129201 20771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:43861 every 8 connection(s)
I20260927 15:17:35.130033 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:17:35.133891 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 20643
I20260927 15:17:35.133994 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:17:35.134284 10186 external_mini_cluster.cc:1428] Setting key a39323b0e5303fd1a164fc3302db06ea
Sep 27 15:17:35 dist-test-slave-m4lt krb5kdc[20402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522255, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:17:35.144208 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.131701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:52627 (local address 127.9.242.190:36557)
0927 15:17:35.131848 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.131853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.132698 (+ 845us) server_negotiation.cc:407] Connection header received
0927 15:17:35.134009 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.134013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.134114 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.134205 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:35.135550 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.136113 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.137647 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.137999 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.138120 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.140628 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:35.140631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:35.140634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:35.140658 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:35.141988 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:35.142602 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:35.142604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:35.142606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:35.142693 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:35.143094 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:35.143097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:35.143099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:35.143306 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:35.143390 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.143909 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.144048 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:17:35.144929 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.132111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36557 (local address 127.9.242.130:52627)
0927 15:17:35.132557 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.132598 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:35.133806 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:35.134360 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:35.134369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:35.134876 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:35.135418 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.135436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.136241 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.136245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.137527 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.137537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.138152 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.138154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.138298 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.138845 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:35.138861 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:35.140450 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:35.142112 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:35.142119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:35.142136 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:35.142472 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:35.142786 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:35.142789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:35.142792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:35.142975 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:35.143393 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:35.143398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:35.143643 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 15:17:35.143802 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 15:17:35.144107 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:17:35.146239 20772 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36557
I20260927 15:17:35.146483 20772 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:35.147011 20772 heartbeater.cc:511] Master 127.9.242.190:36557 requested a full tablet report, sending...
I20260927 15:17:35.148204 20443 ts_manager.cc:194] Registered new tserver with Master: 858199e80a6e4a18b968d555e1a72c8e (127.9.242.130:43861)
I20260927 15:17:35.148914 20443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:52627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:17:35.160440 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36557
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:17:35.268543 20777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:17:35.268810 20777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:17:35.268867 20777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:17:35.273104 20777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:17:35.273171 20777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:17:35.273252 20777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:17:35.278597 20777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:17:35.279619 20777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:17:35.280625 20777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:17:35.289939 20783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:17:35.290009 20782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:35.290095 20777 server_base.cc:1034] running on GCE node
W20260927 15:17:35.289937 20785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:17:35.290464 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:17:35.291208 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:17:35.292392 20777 hybrid_clock.cc:648] HybridClock initialized: now 1790522255292376 us; error 38 us; skew 500 ppm
Sep 27 15:17:35 dist-test-slave-m4lt krb5kdc[20402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522255, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:17:35.295071 20777 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:17:35.295914 20777 webserver.cc:460] Webserver started at http://127.9.242.131:44825/ using document root <none> and password file <none>
I20260927 15:17:35.296442 20777 fs_manager.cc:359] Metadata directory not provided
I20260927 15:17:35.296502 20777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:17:35.296684 20777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:17:35.298707 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "90ad1d6bfad84c778b23188356de47b8"
format_stamp: "Formatted at 2026-09-27 15:17:35 on dist-test-slave-m4lt"
server_key: "375662609ab70b2e0d36196a0fa61ded"
server_key_iv: "505edc440e507424b0d5acd5f02e341a"
server_key_version: "encryptionkey@0"
I20260927 15:17:35.299232 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "90ad1d6bfad84c778b23188356de47b8"
format_stamp: "Formatted at 2026-09-27 15:17:35 on dist-test-slave-m4lt"
server_key: "375662609ab70b2e0d36196a0fa61ded"
server_key_iv: "505edc440e507424b0d5acd5f02e341a"
server_key_version: "encryptionkey@0"
I20260927 15:17:35.302770 20777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:17:35.305052 20792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:35.306389 20777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:17:35.306499 20777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "90ad1d6bfad84c778b23188356de47b8"
format_stamp: "Formatted at 2026-09-27 15:17:35 on dist-test-slave-m4lt"
server_key: "375662609ab70b2e0d36196a0fa61ded"
server_key_iv: "505edc440e507424b0d5acd5f02e341a"
server_key_version: "encryptionkey@0"
I20260927 15:17:35.306597 20777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:17:35.328605 20777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:17:35.332293 20777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:17:35.332471 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:17:35.333019 20777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:17:35.333856 20777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:17:35.333914 20777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:35.334052 20777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:17:35.334082 20777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:17:35.343538 20777 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:43909
I20260927 15:17:35.343556 20905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:43909 every 8 connection(s)
I20260927 15:17:35.344475 20777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:17:35.346271 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 20777
I20260927 15:17:35.346382 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:17:35.346652 10186 external_mini_cluster.cc:1428] Setting key 1d7c484ab09d2104271c3340258c37c7
Sep 27 15:17:35 dist-test-slave-m4lt krb5kdc[20402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522255, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:17:35.357595 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.346293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:60691 (local address 127.9.242.190:36557)
0927 15:17:35.346469 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.346474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.347337 (+ 863us) server_negotiation.cc:407] Connection header received
0927 15:17:35.348720 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.348723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.348775 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.348843 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:35.350032 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.350565 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.351664 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.351882 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.351967 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.354285 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:35.354289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:35.354292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:35.354318 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:35.355609 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:35.356100 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:35.356104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:35.356105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:35.356182 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:35.356478 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:35.356481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:35.356483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:35.356693 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:35.356779 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.357307 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.357424 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 15:17:35.358369 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.346715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36557 (local address 127.9.242.131:60691)
0927 15:17:35.347181 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.347233 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:35.348529 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:35.349018 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:35.349025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:35.349379 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:17:35.349907 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.349921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.350676 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.350679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.351556 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.351565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.352019 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.352023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.352172 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.352673 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:17:35.352687 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:17:35.354139 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:17:35.355723 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:35.355729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:35.355744 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:35.355999 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:35.356260 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:17:35.356263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:17:35.356265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:17:35.356395 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:17:35.356790 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:17:35.356796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:17:35.357081 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 15:17:35.357247 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 15:17:35.357554 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260927 15:17:35.359436 20906 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36557
I20260927 15:17:35.359660 20906 heartbeater.cc:464] Registering TS with master...
I20260927 15:17:35.360163 20906 heartbeater.cc:511] Master 127.9.242.190:36557 requested a full tablet report, sending...
I20260927 15:17:35.361167 20447 ts_manager.cc:194] Registered new tserver with Master: 90ad1d6bfad84c778b23188356de47b8 (127.9.242.131:43909)
I20260927 15:17:35.361821 20447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:60691
I20260927 15:17:35.371621 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:17:35.380234 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.373376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.9.242.190:36557)
0927 15:17:35.373508 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.373512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.373728 (+ 216us) server_negotiation.cc:407] Connection header received
0927 15:17:35.373866 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.373869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.373913 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.373980 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:17:35.374590 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.375126 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.375910 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.376165 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.376241 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.377484 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:17:35.377487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:17:35.377489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:17:35.377512 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:17:35.378822 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:35.379237 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:35.379240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:35.379241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:35.379317 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:17:35.379527 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:17:35.379530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:17:35.379531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:17:35.379737 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:17:35.379835 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.379982 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.380117 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:17:35.383119 20447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:17:35.386021 20447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:17:35.399551 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.393885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:43861 (local address 127.0.0.1:54406)
0927 15:17:35.394199 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.394253 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:35.394374 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:35.395566 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:35.395571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:35.395592 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:35.395724 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.395730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.396740 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.396743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.398048 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.398058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.399046 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.399049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.399145 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.399178 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 15:17:35.399283 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 15:17:35.399396 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260927 15:17:35.399713 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.394607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.9.242.129:42583)
0927 15:17:35.395262 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.395276 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.395299 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:17:35.395371 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.395376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.395581 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.395726 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:35.395906 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.397047 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.398434 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.399199 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.399297 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.399425 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.399481 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.399569 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":107,"threads_started":1}
I20260927 15:17:35.399969 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.394182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:42583 (local address 127.0.0.1:39414)
0927 15:17:35.394767 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.394780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:35.394868 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:35.395731 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:35.395734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:35.395748 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:35.395817 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.395823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.397182 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.397186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.398332 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.398339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.399304 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.399308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.399385 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.399403 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:17:35.399453 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 15:17:35.399504 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":18,"spinlock_wait_cycles":3200,"thread_start_us":89,"threads_started":1}
I20260927 15:17:35.399983 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.394552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43909 (local address 127.0.0.1:45992)
0927 15:17:35.394768 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.394780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:35.394868 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:35.395774 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:35.395777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:35.395793 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:35.395878 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.395883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.397306 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.397309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.398634 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.398644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.399651 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.399655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.399734 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.399757 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:17:35.399805 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:17:35.399866 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":23,"spinlock_wait_cycles":2688,"thread_start_us":78,"threads_started":1}
I20260927 15:17:35.400403 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.394999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.9.242.131:43909)
0927 15:17:35.395409 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.395419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.395436 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:17:35.395494 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.395498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.395639 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.396013 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:35.396165 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.397193 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.398773 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.399527 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.400046 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.400182 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.400228 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.400293 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":138,"threads_started":1}
I20260927 15:17:35.400558 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.394092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.9.242.130:43861)
0927 15:17:35.395100 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.395114 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.395137 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:17:35.395211 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.395216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.395427 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.395564 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:35.395843 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.396620 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.398155 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.398934 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.399070 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.399218 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.399963 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.400421 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":155,"threads_started":1}
I20260927 15:17:35.402909 20572 tablet_service.cc:1467] Processing CreateTablet for tablet bdcc1e6cf32544d6b810bf91768ca9ca (DEFAULT_TABLE table=test-table [id=13c93074c2d6414bbf0c1a0ec910bc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:35.403040 20840 tablet_service.cc:1467] Processing CreateTablet for tablet bdcc1e6cf32544d6b810bf91768ca9ca (DEFAULT_TABLE table=test-table [id=13c93074c2d6414bbf0c1a0ec910bc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:35.403040 20706 tablet_service.cc:1467] Processing CreateTablet for tablet bdcc1e6cf32544d6b810bf91768ca9ca (DEFAULT_TABLE table=test-table [id=13c93074c2d6414bbf0c1a0ec910bc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:17:35.403880 20572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcc1e6cf32544d6b810bf91768ca9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:35.404031 20706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcc1e6cf32544d6b810bf91768ca9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:35.404038 20840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcc1e6cf32544d6b810bf91768ca9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:17:35.410676 20923 tablet_bootstrap.cc:492] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef: Bootstrap starting.
I20260927 15:17:35.411418 20925 tablet_bootstrap.cc:492] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Bootstrap starting.
I20260927 15:17:35.412642 20923 tablet_bootstrap.cc:654] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:35.413381 20925 tablet_bootstrap.cc:654] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:35.413790 20923 log.cc:824] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:35.413997 20924 tablet_bootstrap.cc:492] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Bootstrap starting.
I20260927 15:17:35.414168 20925 log.cc:824] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:35.415697 20923 tablet_bootstrap.cc:492] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef: No bootstrap required, opened a new log
I20260927 15:17:35.415920 20923 ts_tablet_manager.cc:1397] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260927 15:17:35.415932 20924 tablet_bootstrap.cc:654] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Neither blocks nor log segments found. Creating new log.
I20260927 15:17:35.416016 20925 tablet_bootstrap.cc:492] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: No bootstrap required, opened a new log
I20260927 15:17:35.416195 20925 ts_tablet_manager.cc:1397] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:17:35.417062 20924 log.cc:824] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:17:35.418958 20924 tablet_bootstrap.cc:492] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: No bootstrap required, opened a new log
I20260927 15:17:35.419124 20924 ts_tablet_manager.cc:1397] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:17:35.419014 20923 raft_consensus.cc:348] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.419261 20923 raft_consensus.cc:374] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:35.419309 20923 raft_consensus.cc:729] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee5c0e4c84a49339e427fe6d14747ef, State: Initialized, Role: FOLLOWER
I20260927 15:17:35.419713 20923 consensus_queue.cc:260] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.419719 20925 raft_consensus.cc:348] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.419957 20925 raft_consensus.cc:374] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:35.420017 20925 raft_consensus.cc:729] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858199e80a6e4a18b968d555e1a72c8e, State: Initialized, Role: FOLLOWER
I20260927 15:17:35.420419 20925 consensus_queue.cc:260] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.420523 20923 ts_tablet_manager.cc:1428] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 15:17:35.420849 20638 heartbeater.cc:503] Master 127.9.242.190:36557 was elected leader, sending a full tablet report...
I20260927 15:17:35.420974 20772 heartbeater.cc:503] Master 127.9.242.190:36557 was elected leader, sending a full tablet report...
I20260927 15:17:35.422286 20925 ts_tablet_manager.cc:1428] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:17:35.422778 20924 raft_consensus.cc:348] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.422978 20924 raft_consensus.cc:374] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:17:35.423029 20924 raft_consensus.cc:729] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ad1d6bfad84c778b23188356de47b8, State: Initialized, Role: FOLLOWER
I20260927 15:17:35.423388 20924 consensus_queue.cc:260] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.423939 20906 heartbeater.cc:503] Master 127.9.242.190:36557 was elected leader, sending a full tablet report...
I20260927 15:17:35.424261 20924 ts_tablet_manager.cc:1428] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:17:35.453271 20929 raft_consensus.cc:482] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:17:35.453481 20929 raft_consensus.cc:504] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.454743 20929 leader_election.cc:288] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 858199e80a6e4a18b968d555e1a72c8e (127.9.242.130:43861), 90ad1d6bfad84c778b23188356de47b8 (127.9.242.131:43909)
I20260927 15:17:35.458549 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.455034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:54155 (local address 127.9.242.130:43861)
0927 15:17:35.455195 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.455201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.455280 (+ 79us) server_negotiation.cc:407] Connection header received
0927 15:17:35.455460 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.455463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.455553 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.455634 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:35.455906 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.456505 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.457606 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.458040 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.458129 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.458180 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.458351 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.458422 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:17:35.458552 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.454946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:43861 (local address 127.9.242.129:54155)
0927 15:17:35.455200 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.455213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:35.455314 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:35.455658 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:35.455660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:35.455681 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:35.455793 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.455801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.456614 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.456617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.457459 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.457468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.458147 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.458151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.458237 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.458260 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:17:35.458320 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:17:35.458376 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
I20260927 15:17:35.458572 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.455447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:39941 (local address 127.9.242.131:43909)
0927 15:17:35.455562 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.455566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.455770 (+ 204us) server_negotiation.cc:407] Connection header received
0927 15:17:35.455853 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.455856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.455905 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.455966 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:17:35.456210 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.456718 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.457695 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.458088 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.458194 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.458240 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.458414 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.458477 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:17:35.458568 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.455321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43909 (local address 127.9.242.129:39941)
0927 15:17:35.455677 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.455690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:17:35.455778 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:17:35.455987 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:17:35.455989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:17:35.456005 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:17:35.456121 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.456127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.456816 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.456818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.457563 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:17:35.457573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.458204 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.458208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:17:35.458290 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.458308 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:17:35.458379 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 15:17:35.458439 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260927 15:17:35.459192 20726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcc1e6cf32544d6b810bf91768ca9ca" candidate_uuid: "bee5c0e4c84a49339e427fe6d14747ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858199e80a6e4a18b968d555e1a72c8e" is_pre_election: true
I20260927 15:17:35.459210 20860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcc1e6cf32544d6b810bf91768ca9ca" candidate_uuid: "bee5c0e4c84a49339e427fe6d14747ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ad1d6bfad84c778b23188356de47b8" is_pre_election: true
I20260927 15:17:35.459450 20726 raft_consensus.cc:2454] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bee5c0e4c84a49339e427fe6d14747ef in term 0.
I20260927 15:17:35.459460 20860 raft_consensus.cc:2454] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bee5c0e4c84a49339e427fe6d14747ef in term 0.
I20260927 15:17:35.459864 20525 leader_election.cc:302] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 858199e80a6e4a18b968d555e1a72c8e, bee5c0e4c84a49339e427fe6d14747ef; no voters:
I20260927 15:17:35.460122 20929 raft_consensus.cc:2781] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:17:35.460193 20929 raft_consensus.cc:482] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:17:35.460222 20929 raft_consensus.cc:3037] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:35.461172 20929 raft_consensus.cc:504] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.461459 20929 leader_election.cc:288] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [CANDIDATE]: Term 1 election: Requested vote from peers 858199e80a6e4a18b968d555e1a72c8e (127.9.242.130:43861), 90ad1d6bfad84c778b23188356de47b8 (127.9.242.131:43909)
I20260927 15:17:35.461772 20726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcc1e6cf32544d6b810bf91768ca9ca" candidate_uuid: "bee5c0e4c84a49339e427fe6d14747ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858199e80a6e4a18b968d555e1a72c8e"
I20260927 15:17:35.461826 20860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcc1e6cf32544d6b810bf91768ca9ca" candidate_uuid: "bee5c0e4c84a49339e427fe6d14747ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ad1d6bfad84c778b23188356de47b8"
I20260927 15:17:35.461874 20726 raft_consensus.cc:3037] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:35.461933 20860 raft_consensus.cc:3037] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:17:35.462867 20860 raft_consensus.cc:2454] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bee5c0e4c84a49339e427fe6d14747ef in term 1.
I20260927 15:17:35.462873 20726 raft_consensus.cc:2454] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bee5c0e4c84a49339e427fe6d14747ef in term 1.
I20260927 15:17:35.463122 20525 leader_election.cc:302] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 858199e80a6e4a18b968d555e1a72c8e, bee5c0e4c84a49339e427fe6d14747ef; no voters:
I20260927 15:17:35.463281 20929 raft_consensus.cc:2781] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:17:35.463480 20929 raft_consensus.cc:686] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 1 LEADER]: Becoming Leader. State: Replica: bee5c0e4c84a49339e427fe6d14747ef, State: Running, Role: LEADER
I20260927 15:17:35.463848 20929 consensus_queue.cc:237] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } }
I20260927 15:17:35.466885 20446 catalog_manager.cc:5495] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef reported cstate change: term changed from 0 to 1, leader changed from <none> to bee5c0e4c84a49339e427fe6d14747ef (127.9.242.129). New cstate: current_term: 1 leader_uuid: "bee5c0e4c84a49339e427fe6d14747ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bee5c0e4c84a49339e427fe6d14747ef" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42583 } health_report { overall_health: HEALTHY } } }
I20260927 15:17:35.476660 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.473610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.9.242.129:42583)
0927 15:17:35.473739 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:17:35.473742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:17:35.473755 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:17:35.473841 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:17:35.473843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:17:35.473891 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:17:35.473970 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:17:35.474147 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.474617 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.475376 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:17:35.475648 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:17:35.475725 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:17:35.475969 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:17:35.476388 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:17:35.476474 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:17:35.476518 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 15:17:35.476560 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:17:35.482272 20860 raft_consensus.cc:1261] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 1 FOLLOWER]: Refusing update from remote peer bee5c0e4c84a49339e427fe6d14747ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:35.482290 20726 raft_consensus.cc:1261] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 1 FOLLOWER]: Refusing update from remote peer bee5c0e4c84a49339e427fe6d14747ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:17:35.482940 20934 consensus_queue.cc:1035] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "90ad1d6bfad84c778b23188356de47b8" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:35.483139 20929 consensus_queue.cc:1035] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "858199e80a6e4a18b968d555e1a72c8e" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:17:35.491588 20938 mvcc.cc:205] Tried to move back new op lower bound from 7333979158448136192 to 7333979158382362624. Current Snapshot: MvccSnapshot[applied={T|T < 7333979158448136192}]
I20260927 15:17:35.491784 20939 mvcc.cc:205] Tried to move back new op lower bound from 7333979158448136192 to 7333979158382362624. Current Snapshot: MvccSnapshot[applied={T|T < 7333979158448136192}]
I20260927 15:17:35.499243 20443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:35.499415 20443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:17:35.501703 20443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fc1b4172e614a16904879f25e9ab375: Sending DeleteTablet for 3 replicas of tablet bdcc1e6cf32544d6b810bf91768ca9ca
I20260927 15:17:35.502521 20572 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcc1e6cf32544d6b810bf91768ca9ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:35 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:39414
I20260927 15:17:35.502550 20840 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcc1e6cf32544d6b810bf91768ca9ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:35 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:45992
I20260927 15:17:35.502697 20706 tablet_service.cc:1514] Processing DeleteTablet for tablet bdcc1e6cf32544d6b810bf91768ca9ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:17:35 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54406
I20260927 15:17:35.503147 20949 tablet_replica.cc:330] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: stopping tablet replica
I20260927 15:17:35.503319 20948 tablet_replica.cc:330] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef: stopping tablet replica
I20260927 15:17:35.503579 20949 raft_consensus.cc:2229] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:35.503595 20948 raft_consensus.cc:2229] T bdcc1e6cf32544d6b810bf91768ca9ca P bee5c0e4c84a49339e427fe6d14747ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:17:35.503844 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 20509
I20260927 15:17:35.504951 20949 raft_consensus.cc:2258] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:35.504979 20947 tablet_replica.cc:330] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: stopping tablet replica
I20260927 15:17:35.505249 20947 raft_consensus.cc:2229] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:17:35.505523 20947 raft_consensus.cc:2258] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:17:35.506462 20949 ts_tablet_manager.cc:1905] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:35.506771 20947 ts_tablet_manager.cc:1905] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:17:35.508502 20949 ts_tablet_manager.cc:1918] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:35.508594 20949 log.cc:1196] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/bdcc1e6cf32544d6b810bf91768ca9ca
I20260927 15:17:35.508847 20949 ts_tablet_manager.cc:1939] T bdcc1e6cf32544d6b810bf91768ca9ca P 858199e80a6e4a18b968d555e1a72c8e: Deleting consensus metadata
I20260927 15:17:35.509362 20947 ts_tablet_manager.cc:1918] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:17:35.509522 20947 log.cc:1196] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/bdcc1e6cf32544d6b810bf91768ca9ca
I20260927 15:17:35.509855 20947 ts_tablet_manager.cc:1939] T bdcc1e6cf32544d6b810bf91768ca9ca P 90ad1d6bfad84c778b23188356de47b8: Deleting consensus metadata
I20260927 15:17:35.509927 20431 catalog_manager.cc:4841] TS 858199e80a6e4a18b968d555e1a72c8e (127.9.242.130:43861): tablet bdcc1e6cf32544d6b810bf91768ca9ca (table test-table [id=13c93074c2d6414bbf0c1a0ec910bc4a]) successfully deleted
W20260927 15:17:35.510958 20432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:42583 (error 108)
I20260927 15:17:35.510994 20431 catalog_manager.cc:4841] TS 90ad1d6bfad84c778b23188356de47b8 (127.9.242.131:43909): tablet bdcc1e6cf32544d6b810bf91768ca9ca (table test-table [id=13c93074c2d6414bbf0c1a0ec910bc4a]) successfully deleted
I20260927 15:17:35.511685 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 20643
I20260927 15:17:35.511929 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:17:35.511638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:42583 (local address 127.0.0.1:39422)
0927 15:17:35.511752 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 15:17:35.511823 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:42583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:17:35.516620 20432 catalog_manager.cc:4584] TS bee5c0e4c84a49339e427fe6d14747ef (127.9.242.129:42583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdcc1e6cf32544d6b810bf91768ca9ca: Network error: Client connection negotiation failed: client connection to 127.9.242.129:42583: connect: Connection refused (error 111)
I20260927 15:17:35.516981 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 20777
I20260927 15:17:35.521381 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 20415
2026-09-27T15:17:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2108 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:17:35 dist-test-slave-m4lt krb5kdc[20953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:17:35 dist-test-slave-m4lt krb5kdc[20953](info): set up 2 sockets
Sep 27 15:17:35 dist-test-slave-m4lt krb5kdc[20953](info): commencing operation
krb5kdc: starting...
W20260927 15:17:36.470367 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:17:36 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:17:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/rangeradmin_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/rangeradmin_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/rangerlookup_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/rangerlookup_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
I20260927 15:17:36.628005 10186 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task93l8wz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/postgres -l logfile start
2026-09-27 15:17:38.970 UTC [20983] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:17:38.970 UTC [20983] LOG: listening on IPv4 address "127.9.242.148", port 33243
2026-09-27 15:17:38.973 UTC [20983] LOG: listening on Unix socket "/tmp/.s.PGSQL.33243"
2026-09-27 15:17:38.999 UTC [20986] LOG: database system was shut down at 2026-09-27 15:17:37 UTC
2026-09-27 15:17:39.003 UTC [20983] LOG: database system is ready to accept connections
I20260927 15:17:39.874269 10186 mini_postgres.cc:96] Postgres bound to 33243
2026-09-27 15:17:39.880 UTC [20994] FATAL: database "slave" does not exist
127.9.242.148:33243 - accepting connections
I20260927 15:17:39.880780 10186 mini_ranger.cc:161] Starting Ranger...
I20260927 15:17:39.895308 10186 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:17:40.367506 10186 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:17:40.367599 10186 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-admin
2026-09-27 15:17:40,625 [I] DB FLAVOR :POSTGRES
2026-09-27 15:17:40,625 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:17:40,625 [I] Checking connection..
2026-09-27 15:17:40,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:17:40,882 [I] Checking connection passed.
2026-09-27 15:17:40,882 [I] --------- Verifying version history table ---------
2026-09-27 15:17:40,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:17:41,136 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:17:41,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:17:41,367 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:17:41,367 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:17:41,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:17:41.608 UTC [21058] WARNING: there is no transaction in progress
2026-09-27 15:17:41,613 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:17:41,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:17:41,836 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:17:41,837 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:17:41,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:17:42,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:17:42,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m4lt.c.gcp-upstream.internal','N') ;"
2026-09-27 15:17:42,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:17:42,812 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:17:42,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:17:43,055 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:17:43,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:17:43,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:17:43,559 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:17:43,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:17:44.472 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.488 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.504 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.520 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.536 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.556 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.608 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.615 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.628 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.637 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.647 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.661 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.670 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.676 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.682 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:44.688 UTC [21166] WARNING: there is no transaction in progress
2026-09-27 15:17:45,277 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:17:45,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:17:45,522 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:17:45,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:17:45,761 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:17:45,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:17:46,012 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:17:46,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:17:46,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:17:46,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m4lt.c.gcp-upstream.internal';"
2026-09-27 15:17:46,695 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:17:46,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:17:46,949 [I] DB_PATCHES have already been applied
I20260927 15:17:46.954173 10186 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-admin:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task93l8wz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:17:46.954232 10186 mini_ranger.cc:193] Using host: 127.9.242.148
I20260927 15:17:46.957283 10186 mini_ranger.cc:227] Ranger admin URL: http://127.9.242.148:58417
Sep 27, 2026 3:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:17:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:17:47.953146 10186 mini_ranger.cc:160] Time spent starting Ranger: real 8.072s user 0.000s sys 0.010s
Sep 27, 2026 3:18:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:18:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:18:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:18:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:18:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:18:07.785058 10186 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/rangerkms_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/rangerkms_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.242.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.242.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:18:07 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522287, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:18:07.866714 10186 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:18:07.876282 10186 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:18:08.659577 10186 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:18:08.659655 10186 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms
I20260927 15:18:08.659749 10186 mini_ranger_kms.cc:227] Using postgres at 127.9.242.148:33243
2026-09-27 15:18:08,829 [I] DB FLAVOR :POSTGRES
2026-09-27 15:18:08,829 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:18:08,829 [I] Checking connection
2026-09-27 15:18:08,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:18:09,080 [I] connection success
2026-09-27 15:18:09,080 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:18:09,080 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:18:09,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:18:09,330 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:18:09,330 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:18:09,331 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:18:09,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:33243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:18:09,604 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:18:10.067668 10186 mini_ranger_kms.cc:310] Created kms service
I20260927 15:18:10.263756 10186 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:18:10.309695 10186 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:18:10 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522287, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:18:10.684806 10186 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:18:10.685092 10186 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms:/tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:18:10.685115 10186 mini_ranger_kms.cc:240] Using host: 127.9.242.148
I20260927 15:18:10.688439 10186 mini_ranger_kms.cc:276] Ranger KMS PID: 21346
I20260927 15:18:10.688568 10186 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.242.148:50687
Sep 27, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:18:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:18:12.722218 10186 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.856s user 0.018s sys 0.134s
I20260927 15:18:12.722360 10186 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:18:12.722481 10186 mini_ranger_kms.cc:401] 127.9.242.148:50687/kms/v1/keys
Sep 27 15:18:12 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.242.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:18:13 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.242.148@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
Sep 27 15:18:13 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.242.148@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
Sep 27 15:18:15 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:18:15.754720 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43955
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:43955
--ranger_config_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:50687/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:15.868641 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:15.868922 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:15.868961 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:15.873003 21397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:18:15.873073 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:15.873092 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:15.873107 21397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:18:15.873134 21397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:18:15.878743 21397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.242.148:50687/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:43955
--ranger_config_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:18:15.879799 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:15.880777 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:18:15.889042 21397 server_base.cc:1034] running on GCE node
W20260927 15:18:15.889081 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:15.889075 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:15.889075 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:15.889910 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:15.890998 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:15.892155 21397 hybrid_clock.cc:648] HybridClock initialized: now 1790522295892140 us; error 29 us; skew 500 ppm
Sep 27 15:18:15 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522295, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:15.898212 21397 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:18:15.899104 21397 webserver.cc:460] Webserver started at http://127.9.242.190:39957/ using document root <none> and password file <none>
I20260927 15:18:15.899614 21397 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:15.899672 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:15.899854 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:18:15 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522295, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:18:15.962483 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "2468c10527e24a8f9aa0d57690cf462c"
format_stamp: "Formatted at 2026-09-27 15:18:15 on dist-test-slave-m4lt"
server_key: "38e2e2a74552fd2348bc986bc4cd8c77"
server_key_iv: "e1e0ce1c17d682fb9a506742b17bcdc9"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:15.963325 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "2468c10527e24a8f9aa0d57690cf462c"
format_stamp: "Formatted at 2026-09-27 15:18:15 on dist-test-slave-m4lt"
server_key: "38e2e2a74552fd2348bc986bc4cd8c77"
server_key_iv: "e1e0ce1c17d682fb9a506742b17bcdc9"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:15.968310 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:18:15.986948 21412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:15.988586 21397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:18:15.988729 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "2468c10527e24a8f9aa0d57690cf462c"
format_stamp: "Formatted at 2026-09-27 15:18:15 on dist-test-slave-m4lt"
server_key: "38e2e2a74552fd2348bc986bc4cd8c77"
server_key_iv: "e1e0ce1c17d682fb9a506742b17bcdc9"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:15.988867 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:16.010167 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:16.013782 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:16.014036 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:16.021212 21397 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:43955
I20260927 15:18:16.021255 21464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:43955 every 8 connection(s)
I20260927 15:18:16.022219 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:18:16.025593 21465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:16.032389 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 21397
I20260927 15:18:16.032514 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:18:16.032761 10186 external_mini_cluster.cc:1428] Setting key 12c8c88d6f78d7096296b241eee7a65d
I20260927 15:18:16.032953 21465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c: Bootstrap starting.
I20260927 15:18:16.035729 21465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:16.036661 21465 log.cc:824] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:16.038934 21465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c: No bootstrap required, opened a new log
Sep 27 15:18:16 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:18:16.043300 21465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468c10527e24a8f9aa0d57690cf462c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43955 } }
I20260927 15:18:16.043702 21465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:16.043792 21465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2468c10527e24a8f9aa0d57690cf462c, State: Initialized, Role: FOLLOWER
I20260927 15:18:16.044293 21465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468c10527e24a8f9aa0d57690cf462c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43955 } }
I20260927 15:18:16.044435 21465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:18:16.044528 21465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:18:16.044669 21465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:16.045733 21465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468c10527e24a8f9aa0d57690cf462c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43955 } }
I20260927 15:18:16.046118 21465 leader_election.cc:302] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2468c10527e24a8f9aa0d57690cf462c; no voters:
I20260927 15:18:16.046300 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:16.034129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.9.242.190:43955)
0927 15:18:16.034518 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:16.034533 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:16.034568 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:18:16.035176 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:16.035193 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:16.035571 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:16.035873 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:16.036644 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:16.037828 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:16.038654 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:16.039159 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:16.039304 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:16.041717 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:16.041722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:16.041735 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:16.041762 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:16.043226 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:16.043631 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:16.043635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:16.043640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:16.043741 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:16.044052 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:16.044054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:16.044056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:16.044482 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:16.044671 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:16.045207 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 15:18:16.045435 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260927 15:18:16.046448 21465 leader_election.cc:288] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:18:16.046809 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:16.047031 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [term 1 LEADER]: Becoming Leader. State: Replica: 2468c10527e24a8f9aa0d57690cf462c, State: Running, Role: LEADER
I20260927 15:18:16.047333 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468c10527e24a8f9aa0d57690cf462c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43955 } }
I20260927 15:18:16.047739 21465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:18:16.048493 21471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2468c10527e24a8f9aa0d57690cf462c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468c10527e24a8f9aa0d57690cf462c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43955 } } }
I20260927 15:18:16.048616 21471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:16.048830 21471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2468c10527e24a8f9aa0d57690cf462c. Latest consensus state: current_term: 1 leader_uuid: "2468c10527e24a8f9aa0d57690cf462c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2468c10527e24a8f9aa0d57690cf462c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43955 } } }
I20260927 15:18:16.048969 21471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c [sys.catalog]: This master's current role is: LEADER
I20260927 15:18:16.049260 21479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:18:16.052270 21479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:18:16.053272 21465 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:18:16.065745 21479 catalog_manager.cc:1347] Generated new cluster ID: d1305815d3274cc1953365b0989728dc
I20260927 15:18:16.065831 21479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:18:16.088662 21479 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:18:16.090094 21479 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:18:16.101382 21479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c: Generated new TSK 0
I20260927 15:18:16.102380 21479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:18:16.829775 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:50687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43955
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:18:16.979580 21510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:16.979835 21510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:16.979876 21510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:16.984047 21510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:16.984115 21510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:16.984194 21510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:18:16.989646 21510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.242.148:50687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:18:16.990765 21510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:16.991760 21510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:17.002008 21518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:17.002172 21510 server_base.cc:1034] running on GCE node
W20260927 15:18:17.003006 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:17.002017 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:17.006271 21510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:17.007050 21510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:17.008240 21510 hybrid_clock.cc:648] HybridClock initialized: now 1790522297008210 us; error 51 us; skew 500 ppm
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:17.011219 21510 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:18:17.012127 21510 webserver.cc:460] Webserver started at http://127.9.242.129:39837/ using document root <none> and password file <none>
I20260927 15:18:17.012704 21510 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:17.012773 21510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:17.013026 21510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:18:17.029551 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "f9ae4998bb2340508822fe5a856fc88b"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "e4d76f4485dbb2db63bdce06e40e5201"
server_key_iv: "8afef8c7b9288d21dcc4e06d1c6d6bdd"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.030315 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "f9ae4998bb2340508822fe5a856fc88b"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "e4d76f4485dbb2db63bdce06e40e5201"
server_key_iv: "8afef8c7b9288d21dcc4e06d1c6d6bdd"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.038632 21510 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 15:18:17.070345 21527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.071764 21510 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260927 15:18:17.071887 21510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "f9ae4998bb2340508822fe5a856fc88b"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "e4d76f4485dbb2db63bdce06e40e5201"
server_key_iv: "8afef8c7b9288d21dcc4e06d1c6d6bdd"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.072026 21510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:17.085299 21510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:17.088721 21510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:17.088966 21510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:17.089810 21510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:17.090891 21510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:17.090955 21510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.091007 21510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:17.091081 21510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 15:18:17.103138 21510 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:43565
I20260927 15:18:17.103231 21642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:43565 every 8 connection(s)
I20260927 15:18:17.104454 21510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:18:17.108731 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 21510
I20260927 15:18:17.108915 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:18:17.109184 10186 external_mini_cluster.cc:1428] Setting key cefd456eaff198f14997e42cce24782b
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:18:17.121811 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.107034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:45635 (local address 127.9.242.190:43955)
0927 15:18:17.107309 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:17.107313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:17.108268 (+ 955us) server_negotiation.cc:407] Connection header received
0927 15:18:17.109496 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:17.109500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:17.109558 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:17.109638 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:17.111350 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.112178 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.113891 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.114235 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.114316 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.117360 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:17.117364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:17.117367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:17.117394 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:17.119256 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.119878 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.119881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.119883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.119979 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.120393 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.120397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.120399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.120638 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:17.120779 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:17.121468 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 15:18:17.121600 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260927 15:18:17.122890 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.107536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43955 (local address 127.9.242.129:45635)
0927 15:18:17.108085 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:17.108160 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:17.109236 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:17.110133 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:17.110142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:17.110514 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:17.111176 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.111198 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.112338 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.112342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.113755 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.113766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.114540 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.114544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.114747 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.115414 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:17.115436 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:17.117155 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:17.119371 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:17.119378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:17.119396 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:17.119768 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:17.120086 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:17.120089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:17.120091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:17.120286 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:17.120808 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:17.120815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:17.121184 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 15:18:17.121415 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 15:18:17.121813 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
I20260927 15:18:17.124420 21643 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43955
I20260927 15:18:17.124691 21643 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:17.125267 21643 heartbeater.cc:511] Master 127.9.242.190:43955 requested a full tablet report, sending...
I20260927 15:18:17.127024 21429 ts_manager.cc:194] Registered new tserver with Master: f9ae4998bb2340508822fe5a856fc88b (127.9.242.129:43565)
I20260927 15:18:17.128539 21429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:45635
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:18:17.153645 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:50687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43955
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:18:17.309623 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:17.310086 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:17.310195 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:17.316380 21649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:17.316468 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:17.316557 21649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:18:17.324668 21649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.242.148:50687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:18:17.326182 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:17.327545 21649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:17.339712 21656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:17.339867 21655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:17.339715 21658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:17.341059 21649 server_base.cc:1034] running on GCE node
I20260927 15:18:17.341619 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:17.342576 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:17.343776 21649 hybrid_clock.cc:648] HybridClock initialized: now 1790522297343739 us; error 48 us; skew 500 ppm
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:17.347678 21649 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:18:17.349274 21649 webserver.cc:460] Webserver started at http://127.9.242.130:32851/ using document root <none> and password file <none>
I20260927 15:18:17.350317 21649 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:17.350461 21649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:17.350771 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:18:17.369000 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "9fd73862d1155a46ad7178aabc8c83f9"
server_key_iv: "aa3f6efcdbf2111d354da990302ea25a"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.369853 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "9fd73862d1155a46ad7178aabc8c83f9"
server_key_iv: "aa3f6efcdbf2111d354da990302ea25a"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.374917 21649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 15:18:17.394433 21665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.396023 21649 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 15:18:17.396162 21649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "9fd73862d1155a46ad7178aabc8c83f9"
server_key_iv: "aa3f6efcdbf2111d354da990302ea25a"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.396327 21649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:17.414755 21649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:17.417624 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:17.417830 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:17.418431 21649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:17.419278 21649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:17.419342 21649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.419402 21649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:17.419430 21649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.434851 21649 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:42529
I20260927 15:18:17.436112 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:18:17.440383 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 21649
I20260927 15:18:17.440491 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:18:17.440783 10186 external_mini_cluster.cc:1428] Setting key b5fd1248fb3f706c875b528096a6a9d3
I20260927 15:18:17.437747 21778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:42529 every 8 connection(s)
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:18:17.463375 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.441639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:37785 (local address 127.9.242.190:43955)
0927 15:18:17.442440 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:17.442444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:17.443562 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 15:18:17.444954 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:17.444958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:17.445010 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:17.445065 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:17.446763 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.447585 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.449330 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.449605 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.449721 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.457566 (+ 7845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:17.457571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:17.457574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:17.457604 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:17.459311 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.459955 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.459959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.459961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.460054 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.460475 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.460478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.460480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.460716 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:17.460798 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:17.463048 (+ 2250us) server_negotiation.cc:299] Negotiation successful
0927 15:18:17.463191 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722}
I20260927 15:18:17.465346 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.440320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43955 (local address 127.9.242.130:37785)
0927 15:18:17.443408 (+ 3088us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:17.443466 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:17.444640 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:17.445505 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:17.445516 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:17.445903 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:17.446593 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.446613 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.447718 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.447721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.449186 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.449208 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.452341 (+ 3133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.452362 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.452526 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.453559 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:17.453582 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:17.457322 (+ 3740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:17.459465 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:17.459471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:17.459490 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:17.459835 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:17.460165 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:17.460168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:17.460170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:17.460367 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:17.460865 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:17.460871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:17.461198 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 15:18:17.463854 (+ 2656us) client_negotiation.cc:240] Negotiation successful
0927 15:18:17.464239 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2854,"thread_start_us":136,"threads_started":1}
I20260927 15:18:17.469161 21779 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43955
I20260927 15:18:17.469416 21779 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:17.470055 21779 heartbeater.cc:511] Master 127.9.242.190:43955 requested a full tablet report, sending...
I20260927 15:18:17.471315 21429 ts_manager.cc:194] Registered new tserver with Master: ad7041aeb80d4ca7ae1cd065bd3157b1 (127.9.242.130:42529)
I20260927 15:18:17.472146 21429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:37785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:18:17.480759 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:50687/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43955
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
W20260927 15:18:17.639031 21784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:18:17.639503 21784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:18:17.639607 21784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:18:17.645709 21784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:18:17.645990 21784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:18:17.646129 21784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:18:17.653910 21784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.242.148:50687/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:18:17.655453 21784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:18:17.656800 21784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:18:17.675825 21792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:17.676357 21789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:18:17.676550 21790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:18:17.678963 21784 server_base.cc:1034] running on GCE node
I20260927 15:18:17.679466 21784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:18:17.680279 21784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:18:17.695815 21784 hybrid_clock.cc:648] HybridClock initialized: now 1790522297695716 us; error 115 us; skew 500 ppm
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:18:17.699687 21784 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:18:17.701020 21784 webserver.cc:460] Webserver started at http://127.9.242.131:37881/ using document root <none> and password file <none>
I20260927 15:18:17.702148 21784 fs_manager.cc:359] Metadata directory not provided
I20260927 15:18:17.702303 21784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:18:17.702596 21784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:18:17.741371 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "83c1caa3c2f44cee90852d79dee3f74f"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "e8c25bd1ae60ab1b9b5ab5555af07dda"
server_key_iv: "961871fca33ebf250e8a71e989b986b7"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.743402 21784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "83c1caa3c2f44cee90852d79dee3f74f"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "e8c25bd1ae60ab1b9b5ab5555af07dda"
server_key_iv: "961871fca33ebf250e8a71e989b986b7"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.753750 21784 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260927 15:18:17.792328 21799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.795457 21784 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260927 15:18:17.795663 21784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "83c1caa3c2f44cee90852d79dee3f74f"
format_stamp: "Formatted at 2026-09-27 15:18:17 on dist-test-slave-m4lt"
server_key: "e8c25bd1ae60ab1b9b5ab5555af07dda"
server_key_iv: "961871fca33ebf250e8a71e989b986b7"
server_key_version: "kuduclusterkey@0"
I20260927 15:18:17.795876 21784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:18:17.832544 21784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:18:17.836784 21784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:18:17.837074 21784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:18:17.837867 21784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:18:17.839000 21784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:18:17.839105 21784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.839198 21784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:18:17.839260 21784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:18:17.852715 21784 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:40083
I20260927 15:18:17.854019 21784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:18:17.855309 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 21784
I20260927 15:18:17.855415 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:18:17.855691 10186 external_mini_cluster.cc:1428] Setting key c2e871fb844a8131b1709f7f70da57f0
I20260927 15:18:17.854959 21912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:40083 every 8 connection(s)
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:18:17.883163 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.859925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:34679 (local address 127.9.242.190:43955)
0927 15:18:17.860043 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:17.860047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:17.870195 (+ 10148us) server_negotiation.cc:407] Connection header received
0927 15:18:17.871461 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:17.871464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:17.871520 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:17.871571 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:17.873244 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.874086 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.875505 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.875841 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.875913 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.879090 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:17.879094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:17.879097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:17.879125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:17.880712 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.881311 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.881314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.881316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.881401 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.881865 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.881869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.881870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.882094 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:17.882164 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:17.882881 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 15:18:17.883001 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:18:17.885038 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.860501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43955 (local address 127.9.242.131:34679)
0927 15:18:17.870015 (+ 9514us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:17.870096 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:17.871191 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:17.872035 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:17.872045 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:17.872367 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:18:17.873063 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.873083 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.874241 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.874245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.875367 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.875376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.876158 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.876163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.876365 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.877155 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:18:17.877178 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:18:17.878897 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:18:17.880849 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:17.880855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:17.880871 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:17.881209 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:17.881520 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:18:17.881523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:18:17.881525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:18:17.881735 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:18:17.882243 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:18:17.882249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:18:17.882578 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 15:18:17.883595 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0927 15:18:17.883962 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9223,"thread_start_us":148,"threads_started":1}
I20260927 15:18:17.886559 21913 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43955
I20260927 15:18:17.886822 21913 heartbeater.cc:464] Registering TS with master...
I20260927 15:18:17.887396 21913 heartbeater.cc:511] Master 127.9.242.190:43955 requested a full tablet report, sending...
I20260927 15:18:17.888589 21429 ts_manager.cc:194] Registered new tserver with Master: 83c1caa3c2f44cee90852d79dee3f74f (127.9.242.131:40083)
I20260927 15:18:17.889343 21429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:34679
I20260927 15:18:17.893298 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:18:17.907815 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.896409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.9.242.190:43955)
0927 15:18:17.896525 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:17.896529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:17.896542 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:18:17.896577 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:17.896580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:17.896625 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:17.896695 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:18:17.898905 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.899644 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.901169 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.901501 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.901608 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.903635 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:18:17.903638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:18:17.903641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:18:17.903669 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:18:17.905275 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.906141 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.906144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.906146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.906232 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:18:17.906538 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:18:17.906542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:18:17.906543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:18:17.906916 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:18:17.907036 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:17.907514 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 15:18:17.907638 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 15:18:17 dist-test-slave-m4lt krb5kdc[20953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522297, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:18:17.915174 21429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:18:17.921078 21429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:18:17.966276 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.956530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:40083 (local address 127.0.0.1:36460)
0927 15:18:17.960672 (+ 4142us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:17.960684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:17.960763 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:17.961424 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:17.961427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:17.961445 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:17.961529 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.961536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.962836 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.962839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.964539 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.964548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.965724 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.965727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.965836 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.965876 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 15:18:17.965978 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:18:17.966102 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4082,"mutex_wait_us":593,"thread_start_us":89,"threads_started":1}
I20260927 15:18:17.966491 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.954559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:42529 (local address 127.0.0.1:37248)
0927 15:18:17.957919 (+ 3360us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:17.958122 (+ 203us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:17.958775 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:17.960345 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:17.960350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:17.960370 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:17.960473 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.960478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.962704 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.962708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.964106 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.964116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.966144 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.966149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.966247 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.966269 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:18:17.966321 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 15:18:17.966385 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3284,"thread_start_us":77,"threads_started":1}
I20260927 15:18:17.967089 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.955357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.9.242.130:42529)
0927 15:18:17.957589 (+ 2232us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:17.957606 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:17.958470 (+ 864us) server_negotiation.cc:407] Connection header received
0927 15:18:17.958919 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:17.958925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:17.959996 (+ 1071us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:17.960154 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:17.960587 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.962588 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.964266 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.965231 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.966576 (+ 1345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.966725 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:17.966792 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 15:18:17.966899 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2125,"thread_start_us":82,"threads_started":1}
I20260927 15:18:17.968086 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.956696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36460 (local address 127.9.242.131:40083)
0927 15:18:17.957691 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:17.957702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:17.960850 (+ 3148us) server_negotiation.cc:407] Connection header received
0927 15:18:17.960929 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:17.960935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:17.961142 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:17.961288 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:17.961674 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.962728 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.964680 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.965412 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.965500 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.965674 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:17.967889 (+ 2215us) server_negotiation.cc:299] Negotiation successful
0927 15:18:17.967979 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":128,"threads_started":1}
I20260927 15:18:17.970323 21713 tablet_service.cc:1467] Processing CreateTablet for tablet a88bbd2574504174b246f057207169c9 (DEFAULT_TABLE table=test-table [id=2433affbbc0d4659985b73204847b4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:17.970845 21847 tablet_service.cc:1467] Processing CreateTablet for tablet a88bbd2574504174b246f057207169c9 (DEFAULT_TABLE table=test-table [id=2433affbbc0d4659985b73204847b4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:17.971509 21713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88bbd2574504174b246f057207169c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:17.971800 21847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88bbd2574504174b246f057207169c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:17.973054 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.954515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:43565 (local address 127.0.0.1:47402)
0927 15:18:17.958508 (+ 3993us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:17.958520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:17.958632 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:17.968860 (+ 10228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:17.968864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:17.968887 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:17.968993 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.968999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.970546 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.970550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.971638 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:17.971648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.972686 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.972689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:17.972784 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.972810 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 15:18:17.972858 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:18:17.972923 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2543,"thread_start_us":109,"threads_started":1}
I20260927 15:18:17.973737 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:17.955348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.9.242.129:43565)
0927 15:18:17.963110 (+ 7762us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:17.963117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:17.963140 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:18:17.963208 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:17.963214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:17.963417 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:17.969159 (+ 5742us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:17.969318 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.970413 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.971788 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:17.972548 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:17.973314 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:17.973450 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:17.973500 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:18:17.973585 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7651,"thread_start_us":89,"threads_started":1}
I20260927 15:18:17.976436 21558 tablet_service.cc:1467] Processing CreateTablet for tablet a88bbd2574504174b246f057207169c9 (DEFAULT_TABLE table=test-table [id=2433affbbc0d4659985b73204847b4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:18:17.977415 21558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88bbd2574504174b246f057207169c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:18:17.984668 21935 tablet_bootstrap.cc:492] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b: Bootstrap starting.
I20260927 15:18:17.985268 21933 tablet_bootstrap.cc:492] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Bootstrap starting.
I20260927 15:18:17.987531 21933 tablet_bootstrap.cc:654] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:17.988004 21935 tablet_bootstrap.cc:654] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:17.988425 21933 log.cc:824] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:17.988852 21935 log.cc:824] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:17.989255 21934 tablet_bootstrap.cc:492] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Bootstrap starting.
I20260927 15:18:17.991875 21934 tablet_bootstrap.cc:654] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Neither blocks nor log segments found. Creating new log.
I20260927 15:18:17.992674 21934 log.cc:824] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:18:17.995091 21934 tablet_bootstrap.cc:492] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: No bootstrap required, opened a new log
I20260927 15:18:17.995150 21933 tablet_bootstrap.cc:492] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: No bootstrap required, opened a new log
I20260927 15:18:17.995359 21934 ts_tablet_manager.cc:1397] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:18:17.995426 21933 ts_tablet_manager.cc:1397] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260927 15:18:17.999631 21934 raft_consensus.cc:348] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.000486 21933 raft_consensus.cc:348] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.000829 21933 raft_consensus.cc:374] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:18.000921 21933 raft_consensus.cc:729] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7041aeb80d4ca7ae1cd065bd3157b1, State: Initialized, Role: FOLLOWER
I20260927 15:18:18.001453 21933 consensus_queue.cc:260] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.002535 21935 tablet_bootstrap.cc:492] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b: No bootstrap required, opened a new log
I20260927 15:18:18.002758 21933 ts_tablet_manager.cc:1428] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260927 15:18:18.002812 21935 ts_tablet_manager.cc:1397] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260927 15:18:18.002977 21934 raft_consensus.cc:374] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:18.003189 21779 heartbeater.cc:503] Master 127.9.242.190:43955 was elected leader, sending a full tablet report...
I20260927 15:18:18.006573 21934 raft_consensus.cc:729] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c1caa3c2f44cee90852d79dee3f74f, State: Initialized, Role: FOLLOWER
I20260927 15:18:18.007481 21934 consensus_queue.cc:260] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.008627 21934 ts_tablet_manager.cc:1428] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 15:18:18.011770 21935 raft_consensus.cc:348] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.012537 21913 heartbeater.cc:503] Master 127.9.242.190:43955 was elected leader, sending a full tablet report...
I20260927 15:18:18.015013 21935 raft_consensus.cc:374] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:18:18.015156 21935 raft_consensus.cc:729] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ae4998bb2340508822fe5a856fc88b, State: Initialized, Role: FOLLOWER
I20260927 15:18:18.015743 21935 consensus_queue.cc:260] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.016999 21935 ts_tablet_manager.cc:1428] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b: Time spent starting tablet: real 0.013s user 0.002s sys 0.004s
I20260927 15:18:18.017419 21643 heartbeater.cc:503] Master 127.9.242.190:43955 was elected leader, sending a full tablet report...
W20260927 15:18:18.108407 21644 tablet.cc:2369] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:18:18.112010 21914 tablet.cc:2369] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:18:18.193714 21780 tablet.cc:2369] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:18:18.245276 21939 raft_consensus.cc:482] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:18:18.245537 21939 raft_consensus.cc:504] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.246723 21939 leader_election.cc:288] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9ae4998bb2340508822fe5a856fc88b (127.9.242.129:43565), 83c1caa3c2f44cee90852d79dee3f74f (127.9.242.131:40083)
I20260927 15:18:18.252893 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.247128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:40259 (local address 127.9.242.129:43565)
0927 15:18:18.247490 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:18.247495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:18.248012 (+ 517us) server_negotiation.cc:407] Connection header received
0927 15:18:18.248213 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:18.248217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:18.248265 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:18.248321 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:18.248814 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.249521 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.251004 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.251580 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.251669 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:18.251733 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:18.252602 (+ 869us) server_negotiation.cc:299] Negotiation successful
0927 15:18:18.252686 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260927 15:18:18.252976 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.247675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:48659 (local address 127.9.242.131:40083)
0927 15:18:18.247784 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:18.247788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:18.248834 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 15:18:18.249047 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:18.249050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:18.249097 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:18.249138 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:18:18.249545 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.250293 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.251694 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.252243 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.252350 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:18.252404 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:18.252695 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 15:18:18.252777 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:18:18.253396 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.247471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:40083 (local address 127.9.242.130:48659)
0927 15:18:18.248750 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:18.248763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:18.248897 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:18.249322 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:18.249324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:18.249340 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:18.249442 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:18.249446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.250437 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.250440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:18.251574 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:18.251582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.252498 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.252501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:18.252586 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:18.252604 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:18:18.253158 (+ 554us) client_negotiation.cc:240] Negotiation successful
0927 15:18:18.253237 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"thread_start_us":118,"threads_started":1}
I20260927 15:18:18.253707 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.247058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:43565 (local address 127.9.242.130:40259)
0927 15:18:18.247932 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:18.247945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:18:18.248095 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:18:18.248472 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:18:18.248475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:18:18.248493 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:18:18.248620 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:18.248690 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.249635 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.249637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:18.250860 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:18:18.250869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.251816 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.251819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:18:18.252469 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:18.252493 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:18:18.253410 (+ 917us) client_negotiation.cc:240] Negotiation successful
0927 15:18:18.253569 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":117,"threads_started":1}
I20260927 15:18:18.254194 21851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bbd2574504174b246f057207169c9" candidate_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c1caa3c2f44cee90852d79dee3f74f" is_pre_election: true
I20260927 15:18:18.254519 21851 raft_consensus.cc:2454] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7041aeb80d4ca7ae1cd065bd3157b1 in term 0.
I20260927 15:18:18.254509 21597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bbd2574504174b246f057207169c9" candidate_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9ae4998bb2340508822fe5a856fc88b" is_pre_election: true
I20260927 15:18:18.254794 21597 raft_consensus.cc:2454] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7041aeb80d4ca7ae1cd065bd3157b1 in term 0.
I20260927 15:18:18.255127 21668 leader_election.cc:302] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c1caa3c2f44cee90852d79dee3f74f, ad7041aeb80d4ca7ae1cd065bd3157b1; no voters:
I20260927 15:18:18.255429 21939 raft_consensus.cc:2781] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:18:18.255537 21939 raft_consensus.cc:482] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:18:18.255579 21939 raft_consensus.cc:3037] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:18.256883 21939 raft_consensus.cc:504] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.257247 21939 leader_election.cc:288] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [CANDIDATE]: Term 1 election: Requested vote from peers f9ae4998bb2340508822fe5a856fc88b (127.9.242.129:43565), 83c1caa3c2f44cee90852d79dee3f74f (127.9.242.131:40083)
I20260927 15:18:18.257622 21597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bbd2574504174b246f057207169c9" candidate_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9ae4998bb2340508822fe5a856fc88b"
I20260927 15:18:18.257727 21851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88bbd2574504174b246f057207169c9" candidate_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c1caa3c2f44cee90852d79dee3f74f"
I20260927 15:18:18.257758 21597 raft_consensus.cc:3037] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:18.257839 21851 raft_consensus.cc:3037] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:18:18.258901 21597 raft_consensus.cc:2454] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7041aeb80d4ca7ae1cd065bd3157b1 in term 1.
I20260927 15:18:18.259068 21851 raft_consensus.cc:2454] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7041aeb80d4ca7ae1cd065bd3157b1 in term 1.
I20260927 15:18:18.259244 21668 leader_election.cc:302] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad7041aeb80d4ca7ae1cd065bd3157b1, f9ae4998bb2340508822fe5a856fc88b; no voters:
I20260927 15:18:18.259464 21939 raft_consensus.cc:2781] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:18:18.259752 21939 raft_consensus.cc:686] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 1 LEADER]: Becoming Leader. State: Replica: ad7041aeb80d4ca7ae1cd065bd3157b1, State: Running, Role: LEADER
I20260927 15:18:18.260118 21939 consensus_queue.cc:237] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } }
I20260927 15:18:18.263903 21428 catalog_manager.cc:5495] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad7041aeb80d4ca7ae1cd065bd3157b1 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad7041aeb80d4ca7ae1cd065bd3157b1" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 42529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 } health_report { overall_health: UNKNOWN } } }
I20260927 15:18:18.329409 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.324844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.9.242.130:42529)
0927 15:18:18.325177 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 15:18:18.325181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:18:18.325194 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:18:18.325291 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:18:18.325294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:18:18.325359 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:18:18.325460 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:18:18.325772 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.326520 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.327738 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:18:18.328046 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:18:18.328140 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:18:18.328461 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:18:18.329005 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:18:18.329120 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:18:18.329194 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:18:18.329263 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260927 15:18:18.336603 21851 raft_consensus.cc:1261] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 1 FOLLOWER]: Refusing update from remote peer ad7041aeb80d4ca7ae1cd065bd3157b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:18.336603 21597 raft_consensus.cc:1261] T a88bbd2574504174b246f057207169c9 P f9ae4998bb2340508822fe5a856fc88b [term 1 FOLLOWER]: Refusing update from remote peer ad7041aeb80d4ca7ae1cd065bd3157b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:18:18.337296 21939 consensus_queue.cc:1035] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ae4998bb2340508822fe5a856fc88b" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:18.337774 21951 consensus_queue.cc:1035] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c1caa3c2f44cee90852d79dee3f74f" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:18:18.348925 21954 mvcc.cc:205] Tried to move back new op lower bound from 7333979333978959872 to 7333979333676978176. Current Snapshot: MvccSnapshot[applied={T|T < 7333979333978959872}]
I20260927 15:18:18.349099 21955 mvcc.cc:205] Tried to move back new op lower bound from 7333979333978959872 to 7333979333676978176. Current Snapshot: MvccSnapshot[applied={T|T < 7333979333978959872}]
I20260927 15:18:18.363188 21429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:18.363382 21429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:18:18.363839 21957 mvcc.cc:205] Tried to move back new op lower bound from 7333979333978959872 to 7333979333676978176. Current Snapshot: MvccSnapshot[applied={T|T < 7333979333978959872}]
I20260927 15:18:18.366021 21429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2468c10527e24a8f9aa0d57690cf462c: Sending DeleteTablet for 3 replicas of tablet a88bbd2574504174b246f057207169c9
I20260927 15:18:18.366892 21713 tablet_service.cc:1514] Processing DeleteTablet for tablet a88bbd2574504174b246f057207169c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:18 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:37248
I20260927 15:18:18.366964 21847 tablet_service.cc:1514] Processing DeleteTablet for tablet a88bbd2574504174b246f057207169c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:18 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:36460
I20260927 15:18:18.367177 21558 tablet_service.cc:1514] Processing DeleteTablet for tablet a88bbd2574504174b246f057207169c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:18:18 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:47402
I20260927 15:18:18.367997 21965 tablet_replica.cc:330] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: stopping tablet replica
I20260927 15:18:18.368256 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 21510
I20260927 15:18:18.368357 21965 raft_consensus.cc:2229] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:18:18.368579 21965 raft_consensus.cc:2258] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:18.369863 21964 tablet_replica.cc:330] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: stopping tablet replica
I20260927 15:18:18.370141 21964 raft_consensus.cc:2229] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:18:18.370570 21964 raft_consensus.cc:2258] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:18:18.371955 21964 ts_tablet_manager.cc:1905] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:18.375100 21964 ts_tablet_manager.cc:1918] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:18.375293 21964 log.cc:1196] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/a88bbd2574504174b246f057207169c9
I20260927 15:18:18.375659 21964 ts_tablet_manager.cc:1939] T a88bbd2574504174b246f057207169c9 P ad7041aeb80d4ca7ae1cd065bd3157b1: Deleting consensus metadata
I20260927 15:18:18.375687 21965 ts_tablet_manager.cc:1905] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:18:18.376842 21413 catalog_manager.cc:4841] TS ad7041aeb80d4ca7ae1cd065bd3157b1 (127.9.242.130:42529): tablet a88bbd2574504174b246f057207169c9 (table test-table [id=2433affbbc0d4659985b73204847b4e3]) successfully deleted
I20260927 15:18:18.378362 21965 ts_tablet_manager.cc:1918] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:18:18.378470 21965 log.cc:1196] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/a88bbd2574504174b246f057207169c9
I20260927 15:18:18.378777 21965 ts_tablet_manager.cc:1939] T a88bbd2574504174b246f057207169c9 P 83c1caa3c2f44cee90852d79dee3f74f: Deleting consensus metadata
I20260927 15:18:18.380048 21415 catalog_manager.cc:4841] TS 83c1caa3c2f44cee90852d79dee3f74f (127.9.242.131:40083): tablet a88bbd2574504174b246f057207169c9 (table test-table [id=2433affbbc0d4659985b73204847b4e3]) successfully deleted
W20260927 15:18:18.383071 21415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:43565 (error 108)
I20260927 15:18:18.383929 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 21649
I20260927 15:18:18.384104 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:18:18.383845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:43565 (local address 127.0.0.1:47414)
0927 15:18:18.383959 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 15:18:18.384025 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 15:18:18.384749 21415 catalog_manager.cc:4584] TS f9ae4998bb2340508822fe5a856fc88b (127.9.242.129:43565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a88bbd2574504174b246f057207169c9: Network error: Client connection negotiation failed: client connection to 127.9.242.129:43565: connect: Connection refused (error 111)
I20260927 15:18:18.389828 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 21784
I20260927 15:18:18.397475 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 21397
I20260927 15:18:18.404444 10186 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:18:18.739424 10186 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:18:18.739565 10186 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:18:19.074262 10186 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:18:19.074 UTC [20983] LOG: received smart shutdown request
2026-09-27 15:18:19.077 UTC [20983] LOG: background worker "logical replication launcher" (PID 20992) exited with exit code 1
2026-09-27 15:18:19.078 UTC [20987] LOG: shutting down
2026-09-27 15:18:19.222 UTC [20983] LOG: database system is shut down
2026-09-27T15:18:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43793 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:18:19 dist-test-slave-m4lt krb5kdc[21976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:18:19 dist-test-slave-m4lt krb5kdc[21976](info): set up 2 sockets
Sep 27 15:18:19 dist-test-slave-m4lt krb5kdc[21976](info): commencing operation
krb5kdc: starting...
W20260927 15:18:20.269497 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:18:20 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:18:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:18:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/rangeradmin_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/rangeradmin_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/rangerlookup_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/rangerlookup_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
I20260927 15:18:20.435945 10186 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task93l8wz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/postgres -l logfile start
2026-09-27 15:18:22.763 UTC [22006] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 15:18:22.763 UTC [22006] LOG: listening on IPv4 address "127.9.242.148", port 55963
2026-09-27 15:18:22.766 UTC [22006] LOG: listening on Unix socket "/tmp/.s.PGSQL.55963"
2026-09-27 15:18:22.785 UTC [22009] LOG: database system was shut down at 2026-09-27 15:18:21 UTC
2026-09-27 15:18:22.788 UTC [22006] LOG: database system is ready to accept connections
I20260927 15:18:23.661301 10186 mini_postgres.cc:96] Postgres bound to 55963
2026-09-27 15:18:23.666 UTC [22017] FATAL: database "slave" does not exist
127.9.242.148:55963 - accepting connections
I20260927 15:18:23.667462 10186 mini_ranger.cc:161] Starting Ranger...
I20260927 15:18:23.681146 10186 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 15:18:24.106032 10186 mini_ranger.cc:87] Created ranger Postgres database
I20260927 15:18:24.106097 10186 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-admin
2026-09-27 15:18:24,363 [I] DB FLAVOR :POSTGRES
2026-09-27 15:18:24,364 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:18:24,364 [I] Checking connection..
2026-09-27 15:18:24,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 15:18:24,618 [I] Checking connection passed.
2026-09-27 15:18:24,619 [I] --------- Verifying version history table ---------
2026-09-27 15:18:24,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:18:24,868 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:18:24,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:18:25,107 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 15:18:25,108 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 15:18:25,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 15:18:25.359 UTC [22081] WARNING: there is no transaction in progress
2026-09-27 15:18:25,365 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 15:18:25,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 15:18:25,589 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 15:18:25,589 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:18:25,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 15:18:25,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 15:18:26,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m4lt.c.gcp-upstream.internal','N') ;"
2026-09-27 15:18:26,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:18:26,576 [I] Table x_portal_user does not exist in database ranger
2026-09-27 15:18:26,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:18:26,803 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 15:18:26,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:18:27,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:18:27,298 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 15:18:27,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 15:18:28.214 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.230 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.245 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.259 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.274 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.291 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.346 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.353 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.368 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.378 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.389 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.404 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.413 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.419 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.426 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:28.432 UTC [22189] WARNING: there is no transaction in progress
2026-09-27 15:18:29,013 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 15:18:29,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 15:18:29,243 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 15:18:29,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 15:18:29,497 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 15:18:29,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 15:18:29,741 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 15:18:29,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 15:18:29,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 15:18:30,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m4lt.c.gcp-upstream.internal';"
2026-09-27 15:18:30,431 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 15:18:30,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 15:18:30,662 [I] DB_PATCHES have already been applied
I20260927 15:18:30.668195 10186 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-admin:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task93l8wz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 15:18:30.668259 10186 mini_ranger.cc:193] Using host: 127.9.242.148
I20260927 15:18:30.671394 10186 mini_ranger.cc:227] Ranger admin URL: http://127.9.242.148:48127
Sep 27, 2026 3:18:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:18:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:18:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:18:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:18:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:18:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:18:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 15:18:31.663170 10186 mini_ranger.cc:160] Time spent starting Ranger: real 7.996s user 0.002s sys 0.006s
Sep 27, 2026 3:18:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:18:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:18:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:18:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:18:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 15:18:53.400431 10186 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.242.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.242.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/rangerkms_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.242.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/rangerkms_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.242.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.242.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.242.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/HTTP_127.9.242.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 15:18:53 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522333, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 15:18:53.482568 10186 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 15:18:53.491654 10186 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 15:18:54.302879 10186 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 15:18:54.302955 10186 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms
I20260927 15:18:54.302971 10186 mini_ranger_kms.cc:227] Using postgres at 127.9.242.148:55963
2026-09-27 15:18:54,456 [I] DB FLAVOR :POSTGRES
2026-09-27 15:18:54,456 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 15:18:54,456 [I] Checking connection
2026-09-27 15:18:54,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 15:18:54,698 [I] connection success
2026-09-27 15:18:54,699 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 15:18:54,699 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 15:18:54,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 15:18:54,927 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 15:18:54,927 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 15:18:54,927 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 15:18:54,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.242.148:55963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 15:18:55,173 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 15:18:55.665947 10186 mini_ranger_kms.cc:310] Created kms service
I20260927 15:18:55.881286 10186 mini_ranger_kms.cc:326] Created kudu user
I20260927 15:18:55.922837 10186 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 15:18:55 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522333, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:18:56.297981 10186 mini_ranger_kms.cc:384] Added ranger policy
I20260927 15:18:56.298125 10186 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms:/tmp/dist-test-task93l8wz/build/debug/bin/postgresql.jar:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task93l8wz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 15:18:56.298146 10186 mini_ranger_kms.cc:240] Using host: 127.9.242.148
I20260927 15:18:56.301366 10186 mini_ranger_kms.cc:276] Ranger KMS PID: 22369
I20260927 15:18:56.301445 10186 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.242.148:57671
Sep 27, 2026 3:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:18:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 15:18:58.313081 10186 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.831s user 0.009s sys 0.127s
I20260927 15:18:58.313225 10186 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 15:18:58.313306 10186 mini_ranger_kms.cc:401] 127.9.242.148:57671/kms/v1/keys
Sep 27 15:18:58 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522338, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.242.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 15:18:58 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522338, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.242.148@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
Sep 27 15:18:59 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522338, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.242.148@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
Sep 27 15:19:01 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:19:01.676703 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45571
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:45571
--ranger_config_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:01.784703 22420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:01.784977 22420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:01.785032 22420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:01.789166 22420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:01.789224 22420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:01.789256 22420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:01.789290 22420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:01.789318 22420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:01.794720 22420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.242.148:57671/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:45571
--ranger_config_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:01.795763 22420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:01.796746 22420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:01.805071 22426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:01.805083 22428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:01.805095 22425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:01.805315 22420 server_base.cc:1034] running on GCE node
I20260927 15:19:01.806103 22420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:01.807260 22420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:01.808431 22420 hybrid_clock.cc:648] HybridClock initialized: now 1790522341808409 us; error 44 us; skew 500 ppm
Sep 27 15:19:01 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522341, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:01.811296 22420 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:01.812242 22420 webserver.cc:460] Webserver started at http://127.9.242.190:44919/ using document root <none> and password file <none>
I20260927 15:19:01.812745 22420 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:01.812798 22420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:01.812979 22420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:19:01 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522341, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:01.866884 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "2fe27249a8044fc6b302dc605d9b5427"
format_stamp: "Formatted at 2026-09-27 15:19:01 on dist-test-slave-m4lt"
server_key: "e1d0a54c07d1d613b788fb703b7e1a83"
server_key_iv: "9e1f593c9da096ce58a6d30b50f894d0"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:01.867719 22420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "2fe27249a8044fc6b302dc605d9b5427"
format_stamp: "Formatted at 2026-09-27 15:19:01 on dist-test-slave-m4lt"
server_key: "e1d0a54c07d1d613b788fb703b7e1a83"
server_key_iv: "9e1f593c9da096ce58a6d30b50f894d0"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:01.872879 22420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 15:19:01.890136 22435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:01.891831 22420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:19:01.891978 22420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "2fe27249a8044fc6b302dc605d9b5427"
format_stamp: "Formatted at 2026-09-27 15:19:01 on dist-test-slave-m4lt"
server_key: "e1d0a54c07d1d613b788fb703b7e1a83"
server_key_iv: "9e1f593c9da096ce58a6d30b50f894d0"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:01.892113 22420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:01.934860 22420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:01.938269 22420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:01.938472 22420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:01.946967 22420 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:45571
I20260927 15:19:01.947055 22487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:45571 every 8 connection(s)
I20260927 15:19:01.948189 22420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:01.951077 22488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:01.954221 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 22420
I20260927 15:19:01.954348 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:19:01.954630 10186 external_mini_cluster.cc:1428] Setting key cbfa8f662dfbfc399da2d15a115430a9
I20260927 15:19:01.957100 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Bootstrap starting.
I20260927 15:19:01.959635 22488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:01.960351 22488 log.cc:824] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:01.962421 22488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: No bootstrap required, opened a new log
Sep 27 15:19:01 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:01.966394 22488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:01.966713 22488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:01.966761 22488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe27249a8044fc6b302dc605d9b5427, State: Initialized, Role: FOLLOWER
I20260927 15:19:01.967267 22488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:01.967427 22488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:01.967510 22488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:01.967623 22488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:01.968763 22488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:01.969106 22488 leader_election.cc:302] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fe27249a8044fc6b302dc605d9b5427; no voters:
I20260927 15:19:01.969503 22488 leader_election.cc:288] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:01.969674 22493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:01.969990 22493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe27249a8044fc6b302dc605d9b5427, State: Running, Role: LEADER
I20260927 15:19:01.970364 22493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:01.971236 22488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:01.971639 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fe27249a8044fc6b302dc605d9b5427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } } }
I20260927 15:19:01.971832 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:01.972052 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fe27249a8044fc6b302dc605d9b5427. Latest consensus state: current_term: 1 leader_uuid: "2fe27249a8044fc6b302dc605d9b5427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } } }
I20260927 15:19:01.972174 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:01.972349 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:01.956168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.9.242.190:45571)
0927 15:19:01.956689 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:01.956710 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:01.956759 (+ 49us) server_negotiation.cc:407] Connection header received
0927 15:19:01.957643 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:01.957717 (+ 74us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:01.958257 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:01.958636 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:01.959606 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:01.961163 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:01.962341 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:01.963036 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:01.963197 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:01.966363 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:01.966370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:01.966389 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:01.966429 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:01.968381 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:01.968962 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:01.968969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:01.968976 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:01.969121 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:01.969479 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:01.969482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:01.969484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:01.970118 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:01.970905 (+ 787us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:01.971054 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 15:19:01.971323 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":140,"threads_started":1}
I20260927 15:19:01.972610 22497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:01.976660 22497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:01.977057 22488 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:19:01.993770 22497 catalog_manager.cc:1347] Generated new cluster ID: 57a036fced5143d6bec100d432f7aef6
I20260927 15:19:01.993860 22497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:02.022826 22497 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:02.024343 22497 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:02.031306 22497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Generated new TSK 0
I20260927 15:19:02.032292 22497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:19:02.787154 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:45571
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:02.951403 22533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:02.951776 22533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:02.951864 22533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:02.958163 22533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:02.958287 22533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:02.958402 22533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:19:02.966054 22533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.242.148:57671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:02.967545 22533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:02.968885 22533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:02.981460 22543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:02.982805 22533 server_base.cc:1034] running on GCE node
W20260927 15:19:02.982954 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:02.983362 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:02.983888 22533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:02.984730 22533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:02.985913 22533 hybrid_clock.cc:648] HybridClock initialized: now 1790522342985888 us; error 41 us; skew 500 ppm
Sep 27 15:19:02 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:02.989641 22533 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:19:02.990867 22533 webserver.cc:460] Webserver started at http://127.9.242.129:43499/ using document root <none> and password file <none>
I20260927 15:19:02.991657 22533 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:02.991809 22533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:02.992110 22533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:19:02 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:03.014345 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "cf08acf68b2d4c2b94522bd090de3b6f"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "5591b188e498b43c9dbfdd0166663af9"
server_key_iv: "1f0f43b25cdfe885be1d48ac100a26df"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.015316 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "cf08acf68b2d4c2b94522bd090de3b6f"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "5591b188e498b43c9dbfdd0166663af9"
server_key_iv: "1f0f43b25cdfe885be1d48ac100a26df"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.020568 22533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 15:19:03.040956 22550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.042712 22533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:03.042830 22533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "cf08acf68b2d4c2b94522bd090de3b6f"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "5591b188e498b43c9dbfdd0166663af9"
server_key_iv: "1f0f43b25cdfe885be1d48ac100a26df"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.042955 22533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:03.062052 22533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:03.065629 22533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:03.065887 22533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:03.066677 22533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:03.067696 22533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:03.067807 22533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.067919 22533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:03.067991 22533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.078588 22533 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:46117
I20260927 15:19:03.080068 22533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:03.085904 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:46117 every 8 connection(s)
I20260927 15:19:03.086408 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 22533
I20260927 15:19:03.086530 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:03.086813 10186 external_mini_cluster.cc:1428] Setting key 7fbb9ba2ceb29e16b795f72b4c4c10d3
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:03.115863 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:03.092198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:39119 (local address 127.9.242.190:45571)
0927 15:19:03.098728 (+ 6530us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:03.098733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:03.102238 (+ 3505us) server_negotiation.cc:407] Connection header received
0927 15:19:03.103510 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:03.103514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:03.103580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:03.103715 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:03.105405 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.106275 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.107932 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.108283 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.108352 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:03.111578 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:03.111582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:03.111585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:03.111614 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:03.113361 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:03.114065 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:03.114068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:03.114070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:03.114164 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:03.114542 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:03.114545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:03.114547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:03.114781 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:03.114858 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:03.115531 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 15:19:03.115681 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6423,"thread_start_us":86,"threads_started":1}
I20260927 15:19:03.117900 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:03.092747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:45571 (local address 127.9.242.129:39119)
0927 15:19:03.102010 (+ 9263us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:03.102104 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:03.103250 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:03.104074 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:03.104085 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:03.104466 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:03.105213 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:03.105235 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.106430 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.106434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:03.107791 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:03.107804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.108564 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.108568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:03.108867 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:03.109529 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:03.109551 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:03.111356 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:03.113516 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:03.113523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:03.113543 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:03.113950 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:03.114266 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:03.114269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:03.114271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:03.114448 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:03.114922 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:03.114928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:03.115262 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 15:19:03.116419 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0927 15:19:03.116795 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8966,"thread_start_us":156,"threads_started":1}
I20260927 15:19:03.119541 22666 heartbeater.cc:347] Connected to a master server at 127.9.242.190:45571
I20260927 15:19:03.119792 22666 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:03.120337 22666 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:03.122120 22452 ts_manager.cc:194] Registered new tserver with Master: cf08acf68b2d4c2b94522bd090de3b6f (127.9.242.129:46117)
I20260927 15:19:03.123633 22452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:39119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:19:03.140738 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:45571
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 15:19:03.327593 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:03.328369 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:03.328498 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:03.334738 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:03.334920 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:03.335057 22672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:19:03.342052 22672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.242.148:57671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:03.343174 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:03.344225 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:03.354864 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:03.355357 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:03.356366 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:03.356469 22672 server_base.cc:1034] running on GCE node
I20260927 15:19:03.356865 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:03.357499 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:03.358672 22672 hybrid_clock.cc:648] HybridClock initialized: now 1790522343358602 us; error 85 us; skew 500 ppm
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:03.362383 22672 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:19:03.363657 22672 webserver.cc:460] Webserver started at http://127.9.242.130:43585/ using document root <none> and password file <none>
I20260927 15:19:03.364460 22672 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:03.364547 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:03.364775 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:03.380153 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "c43b753ef32c459a878c43577a567ea8"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "4d0d3e73fd680e107d07b055075c1531"
server_key_iv: "4c3771cffb7c4caea0612ae9874d7f9a"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.381886 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "c43b753ef32c459a878c43577a567ea8"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "4d0d3e73fd680e107d07b055075c1531"
server_key_iv: "4c3771cffb7c4caea0612ae9874d7f9a"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.391335 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260927 15:19:03.417472 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.425851 22672 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260927 15:19:03.426012 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "c43b753ef32c459a878c43577a567ea8"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "4d0d3e73fd680e107d07b055075c1531"
server_key_iv: "4c3771cffb7c4caea0612ae9874d7f9a"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.426163 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:03.472527 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:03.478758 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:03.479017 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:03.479775 22672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:03.480854 22672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:03.480944 22672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.481024 22672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:03.481073 22672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.491546 22672 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:34615
I20260927 15:19:03.492710 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:03.498245 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 22672
I20260927 15:19:03.498370 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:03.498656 10186 external_mini_cluster.cc:1428] Setting key 67271459d742243a572d9a7f2d763f1b
I20260927 15:19:03.501072 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:34615 every 8 connection(s)
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:03.515386 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:03.499450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:42115 (local address 127.9.242.190:45571)
0927 15:19:03.499581 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:03.499586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:03.502202 (+ 2616us) server_negotiation.cc:407] Connection header received
0927 15:19:03.503348 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:03.503352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:03.503408 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:03.503494 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:03.505468 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.506273 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.507927 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.508340 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.508447 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:03.511110 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:03.511114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:03.511118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:03.511146 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:03.512877 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:03.513482 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:03.513485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:03.513487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:03.513578 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:03.514094 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:03.514097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:03.514099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:03.514337 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:03.514408 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:03.515079 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 15:19:03.515209 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:19:03.517167 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:03.500020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:45571 (local address 127.9.242.130:42115)
0927 15:19:03.501991 (+ 1971us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:03.502050 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:03.503081 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:03.504282 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:03.504292 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:03.504642 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:03.505322 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:03.505341 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.506394 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.506398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:03.507796 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:03.507807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.508507 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.508512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:03.508711 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:03.509362 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:03.509382 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:03.510935 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:03.513011 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:03.513017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:03.513033 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:03.513372 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:03.513778 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:03.513781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:03.513783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:03.513981 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:03.514486 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:03.514492 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:03.514813 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 15:19:03.515796 (+ 983us) client_negotiation.cc:240] Negotiation successful
0927 15:19:03.516130 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1737,"thread_start_us":141,"threads_started":1}
I20260927 15:19:03.518738 22802 heartbeater.cc:347] Connected to a master server at 127.9.242.190:45571
I20260927 15:19:03.518989 22802 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:03.519560 22802 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:19:03.520696 22452 ts_manager.cc:194] Registered new tserver with Master: c43b753ef32c459a878c43577a567ea8 (127.9.242.130:34615)
I20260927 15:19:03.521643 22452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:42115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:19:03.553222 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:45571
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
W20260927 15:19:03.719637 22807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:03.721717 22807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:03.721870 22807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:03.728652 22807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:03.728878 22807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:03.729029 22807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:19:03.737337 22807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.242.148:57671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:03.738974 22807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:03.740360 22807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:03.767688 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:03.767876 22815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:03.768812 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:03.769371 22807 server_base.cc:1034] running on GCE node
I20260927 15:19:03.769965 22807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:03.770864 22807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:03.772050 22807 hybrid_clock.cc:648] HybridClock initialized: now 1790522343771977 us; error 88 us; skew 500 ppm
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:03.775779 22807 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:19:03.776883 22807 webserver.cc:460] Webserver started at http://127.9.242.131:37249/ using document root <none> and password file <none>
I20260927 15:19:03.777566 22807 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:03.777673 22807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:03.777915 22807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:03.800640 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "8e6144c750aa50affc4d54a89ab7aeb8"
server_key_iv: "60e3ef47067dfbd95d2d3e17d8cce865"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.801476 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "8e6144c750aa50affc4d54a89ab7aeb8"
server_key_iv: "60e3ef47067dfbd95d2d3e17d8cce865"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.806789 22807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260927 15:19:03.830122 22822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.831990 22807 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260927 15:19:03.832130 22807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "8e6144c750aa50affc4d54a89ab7aeb8"
server_key_iv: "60e3ef47067dfbd95d2d3e17d8cce865"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:03.832280 22807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:03.875432 22807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:03.879097 22807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:03.879402 22807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:03.880107 22807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:03.881198 22807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:03.881312 22807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.881410 22807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:03.881470 22807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:03.893894 22807 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:45443
I20260927 15:19:03.895138 22807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:03.896003 22935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:45443 every 8 connection(s)
I20260927 15:19:03.896426 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 22807
I20260927 15:19:03.896543 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:03.896816 10186 external_mini_cluster.cc:1428] Setting key a44b6eed7a807a85d6677e82b09d8492
Sep 27 15:19:03 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522343, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:03.913604 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:03.900299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:36151 (local address 127.9.242.190:45571)
0927 15:19:03.900413 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:03.900417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:03.900877 (+ 460us) server_negotiation.cc:407] Connection header received
0927 15:19:03.902070 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:03.902073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:03.902135 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:03.902182 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:03.903753 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.904580 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.906264 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.906609 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.906680 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:03.909625 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:03.909629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:03.909632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:03.909691 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:03.911290 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:03.911895 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:03.911898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:03.911900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:03.911985 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:03.912372 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:03.912375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:03.912377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:03.912587 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:03.912652 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:03.913324 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 15:19:03.913444 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:19:03.915609 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:03.899233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:45571 (local address 127.9.242.131:36151)
0927 15:19:03.900733 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:03.900790 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:03.901840 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:03.902574 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:03.902583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:03.902934 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:03.903582 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:03.903616 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.904727 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.904730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:03.906132 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:03.906142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:03.906879 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:03.906883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:03.907076 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:03.907735 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:03.907755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:03.909426 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:03.911427 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:03.911434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:03.911451 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:03.911786 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:03.912082 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:03.912085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:03.912087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:03.912273 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:03.912725 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:03.912732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:03.913048 (+ 316us) client_negotiation.cc:769] Sending connection context
0927 15:19:03.914211 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0927 15:19:03.914563 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":150,"threads_started":1}
I20260927 15:19:03.917060 22936 heartbeater.cc:347] Connected to a master server at 127.9.242.190:45571
I20260927 15:19:03.917301 22936 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:03.917965 22936 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:03.919078 22452 ts_manager.cc:194] Registered new tserver with Master: 1ac819aa12914c6ab7e6bdea45b9a5ca (127.9.242.131:45443)
I20260927 15:19:03.919788 22452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:36151
I20260927 15:19:03.922912 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:03.923758 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 22533
I20260927 15:19:03.932597 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 22672
I20260927 15:19:03.939067 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 22807
I20260927 15:19:03.945988 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 22420
I20260927 15:19:03.951300 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45571
--webserver_interface=127.9.242.190
--webserver_port=44919
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:45571
--ranger_config_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:04.056772 22939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:04.057034 22939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:04.057072 22939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:04.061033 22939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:04.061092 22939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:04.061110 22939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:04.061125 22939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:04.061138 22939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:04.066569 22939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:45571
--ranger_config_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:04.067597 22939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:04.068593 22939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:04.077194 22945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.077194 22944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.077250 22947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:04.077584 22939 server_base.cc:1034] running on GCE node
I20260927 15:19:04.078269 22939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:04.079396 22939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:04.080579 22939 hybrid_clock.cc:648] HybridClock initialized: now 1790522344080563 us; error 34 us; skew 500 ppm
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:04.083362 22939 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:04.084365 22939 webserver.cc:460] Webserver started at http://127.9.242.190:44919/ using document root <none> and password file <none>
I20260927 15:19:04.084919 22939 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:04.084991 22939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:04.143276 22939 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:19:04.146055 22955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.147673 22939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:04.147814 22939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "2fe27249a8044fc6b302dc605d9b5427"
format_stamp: "Formatted at 2026-09-27 15:19:01 on dist-test-slave-m4lt"
server_key: "e1d0a54c07d1d613b788fb703b7e1a83"
server_key_iv: "9e1f593c9da096ce58a6d30b50f894d0"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:04.148514 22939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:04.161875 22939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:04.165053 22939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:04.165251 22939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:04.173549 22939 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:45571
I20260927 15:19:04.173560 23007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:45571 every 8 connection(s)
I20260927 15:19:04.174942 22939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:04.177838 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 22939
I20260927 15:19:04.178319 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:46117
--local_ip_for_outbound_sockets=127.9.242.129
--tserver_master_addrs=127.9.242.190:45571
--webserver_port=43499
--webserver_interface=127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:04.179776 23008 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:19:04.181397 23008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Bootstrap starting.
I20260927 15:19:04.203735 23008 log.cc:824] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:04.211849 23008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:04.212311 23008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Bootstrap complete.
I20260927 15:19:04.217742 23008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:04.218253 23008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe27249a8044fc6b302dc605d9b5427, State: Initialized, Role: FOLLOWER
I20260927 15:19:04.218802 23008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:04.218943 23008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:04.219007 23008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:04.219097 23008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:19:04.221704 23008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:04.222080 23008 leader_election.cc:302] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fe27249a8044fc6b302dc605d9b5427; no voters:
I20260927 15:19:04.222430 23008 leader_election.cc:288] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:19:04.222572 23012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:19:04.222852 23012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [term 2 LEADER]: Becoming Leader. State: Replica: 2fe27249a8044fc6b302dc605d9b5427, State: Running, Role: LEADER
I20260927 15:19:04.223192 23012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } }
I20260927 15:19:04.223724 23008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:04.224546 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fe27249a8044fc6b302dc605d9b5427. Latest consensus state: current_term: 2 leader_uuid: "2fe27249a8044fc6b302dc605d9b5427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } } }
I20260927 15:19:04.224670 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:04.224586 23013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2fe27249a8044fc6b302dc605d9b5427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe27249a8044fc6b302dc605d9b5427" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45571 } } }
I20260927 15:19:04.224757 23013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:04.225014 23016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:04.228181 23016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:04.228691 23008 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 15:19:04.228701 23016 catalog_manager.cc:1259] Loaded cluster ID: 57a036fced5143d6bec100d432f7aef6
I20260927 15:19:04.228976 23016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:04.234102 23016 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:04.235522 23016 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Loaded TSK: 0
I20260927 15:19:04.236974 23016 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:19:04.318039 23010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:04.318296 23010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:04.318353 23010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:04.322999 23010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:04.323063 23010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:04.323143 23010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:19:04.328768 23010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:46117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=43499
--tserver_master_addrs=127.9.242.190:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:04.330163 23010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:04.331509 23010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:04.343304 23045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.343320 23048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.343674 23046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:04.344199 23010 server_base.cc:1034] running on GCE node
I20260927 15:19:04.344631 23010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:04.345376 23010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:04.346573 23010 hybrid_clock.cc:648] HybridClock initialized: now 1790522344346549 us; error 46 us; skew 500 ppm
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:04.349970 23010 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:19:04.351035 23010 webserver.cc:460] Webserver started at http://127.9.242.129:43499/ using document root <none> and password file <none>
I20260927 15:19:04.351719 23010 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:04.351806 23010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:04.389380 23010 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260927 15:19:04.392866 23056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.394579 23010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 15:19:04.394726 23010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "cf08acf68b2d4c2b94522bd090de3b6f"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "5591b188e498b43c9dbfdd0166663af9"
server_key_iv: "1f0f43b25cdfe885be1d48ac100a26df"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:04.395510 23010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:04.428395 23010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:04.432261 23010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:04.432475 23010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:04.433240 23010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:04.434355 23010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:04.434414 23010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.434475 23010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:04.434504 23010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.445853 23010 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:46117
I20260927 15:19:04.447214 23010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:04.450595 23169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:46117 every 8 connection(s)
I20260927 15:19:04.451556 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23010
I20260927 15:19:04.452067 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:34615
--local_ip_for_outbound_sockets=127.9.242.130
--tserver_master_addrs=127.9.242.190:45571
--webserver_port=43585
--webserver_interface=127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:04.477614 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:04.452456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:53895 (local address 127.9.242.190:45571)
0927 15:19:04.454859 (+ 2403us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:04.454877 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:04.454913 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:19:04.455138 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:04.455149 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:04.455484 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:04.455810 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:04.462656 (+ 6846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.464195 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.465959 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.466763 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.466897 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:04.471545 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:04.471552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:04.471570 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:04.471603 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:04.473625 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.474293 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.474300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.474306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.474444 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.474805 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.474808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.474810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.475275 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:04.475388 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:04.476255 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 15:19:04.476584 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2253,"thread_start_us":137,"threads_started":1}
I20260927 15:19:04.479264 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:04.452611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:45571 (local address 127.9.242.129:53895)
0927 15:19:04.453346 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:04.453399 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:04.454406 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:04.455881 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:04.455889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:04.461586 (+ 5697us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:04.462419 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:04.462442 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.464371 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.464375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:04.465816 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:04.465827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.467189 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.467193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:04.467395 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:04.468450 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:04.468509 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:04.471297 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:04.473791 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:04.473797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:04.473813 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:04.474151 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:04.474547 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:04.474550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:04.474552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:04.474708 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:04.475573 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:04.475580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:04.475907 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 15:19:04.478082 (+ 2175us) client_negotiation.cc:240] Negotiation successful
0927 15:19:04.478402 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":134,"threads_started":1}
I20260927 15:19:04.481954 23170 heartbeater.cc:347] Connected to a master server at 127.9.242.190:45571
I20260927 15:19:04.482193 23170 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.482748 23170 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
W20260927 15:19:04.484889 23170 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:45571 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:19:04.485011 23170 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.485190 23170 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:04.486003 23170 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.486171 23170 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
W20260927 15:19:04.486847 23170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:19:04.595482 23174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:04.595752 23174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:04.595795 23174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:04.600942 23174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:04.601004 23174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:04.601064 23174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:19:04.607472 23174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:34615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=43585
--tserver_master_addrs=127.9.242.190:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:04.608580 23174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:04.609699 23174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:04.621136 23182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.621342 23179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.621966 23180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:04.622159 23174 server_base.cc:1034] running on GCE node
I20260927 15:19:04.622663 23174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:04.623478 23174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:04.624694 23174 hybrid_clock.cc:648] HybridClock initialized: now 1790522344624657 us; error 48 us; skew 500 ppm
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:04.627776 23174 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:19:04.628894 23174 webserver.cc:460] Webserver started at http://127.9.242.130:43585/ using document root <none> and password file <none>
I20260927 15:19:04.629459 23174 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:04.629539 23174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:04.651953 23174 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.002s
I20260927 15:19:04.660146 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.661890 23174 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.001s
I20260927 15:19:04.662039 23174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "c43b753ef32c459a878c43577a567ea8"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "4d0d3e73fd680e107d07b055075c1531"
server_key_iv: "4c3771cffb7c4caea0612ae9874d7f9a"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:04.662803 23174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:04.687384 23174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:04.690330 23174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:04.690508 23174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:04.691104 23174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:04.691911 23174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:04.691958 23174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.692008 23174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:04.692034 23174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.701632 23174 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:34615
I20260927 15:19:04.702988 23174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:04.703476 23303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:34615 every 8 connection(s)
I20260927 15:19:04.704342 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23174
I20260927 15:19:04.704794 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.242.148:57671/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:45443
--local_ip_for_outbound_sockets=127.9.242.131
--tserver_master_addrs=127.9.242.190:45571
--webserver_port=37249
--webserver_interface=127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:04.720744 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:04.707742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:58685 (local address 127.9.242.190:45571)
0927 15:19:04.707898 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:04.707903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:04.707925 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:19:04.709046 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:04.709050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:04.709122 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:04.709181 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:04.710898 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.711716 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.713435 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.713808 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.713890 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:04.716700 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:04.716705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:04.716708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:04.716739 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:04.718409 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.719005 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.719009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.719011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.719092 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.719486 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.719489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.719491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.719705 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:04.719770 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:04.720437 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 15:19:04.720573 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:19:04.722615 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:04.706649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:45571 (local address 127.9.242.130:58685)
0927 15:19:04.707741 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:04.707799 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:04.708797 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:04.709676 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:04.709687 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:04.710031 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:04.710717 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:04.710737 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.711876 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.711880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:04.713289 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:04.713300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.714103 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.714108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:04.714331 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:04.714943 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:04.714967 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:04.716497 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:04.718539 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:04.718546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:04.718564 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:04.718896 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:04.719207 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:04.719210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:04.719212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:04.719387 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:04.719842 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:04.719849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:04.720172 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 15:19:04.721203 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0927 15:19:04.721552 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":114,"threads_started":1}
I20260927 15:19:04.724181 23304 heartbeater.cc:347] Connected to a master server at 127.9.242.190:45571
I20260927 15:19:04.724448 23304 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.725075 23304 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
W20260927 15:19:04.726995 23304 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:45571 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:19:04.727129 23304 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.727308 23304 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:04.728183 23304 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.728358 23304 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
W20260927 15:19:04.729086 23304 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:19:04.857723 23306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:04.858013 23306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:04.858074 23306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:04.863049 23306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:04.863116 23306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:04.863194 23306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:19:04.869163 23306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.242.148:57671/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:45443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=37249
--tserver_master_addrs=127.9.242.190:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:04.870579 23306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:04.871773 23306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:04.882100 23313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.882032 23316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:04.882030 23314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:04.883193 23306 server_base.cc:1034] running on GCE node
I20260927 15:19:04.883652 23306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:04.884306 23306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:04.885481 23306 hybrid_clock.cc:648] HybridClock initialized: now 1790522344885453 us; error 38 us; skew 500 ppm
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:04.888181 23306 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:19:04.889183 23306 webserver.cc:460] Webserver started at http://127.9.242.131:37249/ using document root <none> and password file <none>
I20260927 15:19:04.889776 23306 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:04.889863 23306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:04.923416 23306 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 15:19:04.928581 23324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.930409 23306 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 15:19:04.930562 23306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca"
format_stamp: "Formatted at 2026-09-27 15:19:03 on dist-test-slave-m4lt"
server_key: "8e6144c750aa50affc4d54a89ab7aeb8"
server_key_iv: "60e3ef47067dfbd95d2d3e17d8cce865"
server_key_version: "kuduclusterkey@0"
I20260927 15:19:04.931346 23306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:04.948088 23306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:04.951859 23306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:04.952064 23306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:04.952795 23306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:04.953878 23306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:04.953943 23306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.954005 23306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:04.954034 23306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:04.964839 23306 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:45443
I20260927 15:19:04.965687 23437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:45443 every 8 connection(s)
I20260927 15:19:04.966133 23306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:04.969961 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23306
Sep 27 15:19:04 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:04.983714 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:04.968364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:59853 (local address 127.9.242.190:45571)
0927 15:19:04.968496 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:04.968500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:04.970504 (+ 2004us) server_negotiation.cc:407] Connection header received
0927 15:19:04.971980 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:04.971983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:04.972020 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:04.972091 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:04.973477 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.974054 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.975758 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.976121 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.976237 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:04.979387 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:04.979391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:04.979393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:04.979444 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:04.980982 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.981686 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.981698 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.981700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.981789 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.982225 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.982229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.982231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.982525 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:04.982612 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:04.983365 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 15:19:04.983508 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:19:04.985610 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:04.969222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:45571 (local address 127.9.242.131:59853)
0927 15:19:04.970346 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:04.970398 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:04.971775 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:04.972289 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:04.972298 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:04.972641 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:04.973328 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:04.973348 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.974205 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.974209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:04.975623 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:04.975634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.976272 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.976275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:04.976476 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:04.977234 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:04.977257 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:04.979198 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:04.981134 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:04.981141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:04.981160 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:04.981539 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:04.981903 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:04.981906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:04.981908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:04.982114 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:04.982706 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:04.982713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:04.983062 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 15:19:04.984183 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0927 15:19:04.984566 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":123,"threads_started":1}
I20260927 15:19:04.987538 23438 heartbeater.cc:347] Connected to a master server at 127.9.242.190:45571
I20260927 15:19:04.987813 23438 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.988443 23438 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
W20260927 15:19:04.990207 23438 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:45571 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 15:19:04.990352 23438 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.990540 23438 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:04.991345 23438 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:04.991516 23438 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:04.991850 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:04.970661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.9.242.190:45571)
0927 15:19:04.971333 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:04.971336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:04.971593 (+ 257us) server_negotiation.cc:407] Connection header received
0927 15:19:04.971707 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:04.971710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:04.971758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:04.971836 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:04.972979 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.973540 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.975617 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:04.975913 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:04.975982 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:04.978931 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:04.978935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:04.978938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:04.978970 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:04.985897 (+ 6927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.989114 (+ 3217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.989119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.989121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.989220 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:04.990006 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:04.990010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:04.990011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:04.990234 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:04.991248 (+ 1014us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:04.991324 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 15:19:04.991505 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":86,"threads_started":1}
W20260927 15:19:04.992569 23438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 15:19:05 dist-test-slave-m4lt krb5kdc[21976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790522345, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:05.487077 23170 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:05.487371 23170 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:05.488546 22966 ts_manager.cc:194] Registered new tserver with Master: cf08acf68b2d4c2b94522bd090de3b6f (127.9.242.129:46117)
I20260927 15:19:05.490183 22966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:53895
I20260927 15:19:05.729336 23304 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:05.729635 23304 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:05.730439 22966 ts_manager.cc:194] Registered new tserver with Master: c43b753ef32c459a878c43577a567ea8 (127.9.242.130:34615)
I20260927 15:19:05.731158 22966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:58685
Sep 27 15:19:05 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522345, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:05.992787 23438 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:05.993099 23438 heartbeater.cc:511] Master 127.9.242.190:45571 requested a full tablet report, sending...
I20260927 15:19:05.994326 22966 ts_manager.cc:194] Registered new tserver with Master: 1ac819aa12914c6ab7e6bdea45b9a5ca (127.9.242.131:45443)
I20260927 15:19:05.995117 22966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:59853
I20260927 15:19:06.010128 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:06.022743 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.011832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.9.242.190:45571)
0927 15:19:06.012092 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:06.012097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:06.012774 (+ 677us) server_negotiation.cc:407] Connection header received
0927 15:19:06.012892 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:06.012896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:06.012954 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:06.013037 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:06.014019 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.014834 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.016029 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.016399 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.016504 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.018349 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:06.018353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:06.018356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:06.018387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:06.020236 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:06.020755 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:06.020758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:06.020760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:06.020863 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:06.021209 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:06.021212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:06.021214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:06.021471 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:06.021602 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:06.022413 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 15:19:06.022558 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260927 15:19:06.027073 22966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:06.031906 22966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 27 15:19:06 dist-test-slave-m4lt krb5kdc[21976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522345, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for HTTP/127.9.242.148@KRBTEST.COM
I20260927 15:19:06.071084 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.062109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:34615 (local address 127.0.0.1:36118)
0927 15:19:06.064057 (+ 1948us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:06.064108 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:06.064207 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:06.065008 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:06.065013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:06.065032 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:06.065131 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.065136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.067325 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.067329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.068439 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.068451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.070485 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.070489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.070673 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.070726 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 15:19:06.070837 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 15:19:06.070956 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"mutex_wait_us":35,"thread_start_us":112,"threads_started":1}
I20260927 15:19:06.071693 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.062217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.9.242.130:34615)
0927 15:19:06.064426 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:06.064432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:06.064449 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:19:06.064507 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:06.064511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:06.064695 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:06.064861 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:06.065238 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.067168 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.068580 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.070178 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.070270 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.070421 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:06.071453 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0927 15:19:06.071558 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2123,"thread_start_us":79,"threads_started":1}
I20260927 15:19:06.075004 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.062862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:45443 (local address 127.0.0.1:53014)
0927 15:19:06.069807 (+ 6945us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:06.069822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:06.070002 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:06.070526 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:06.070529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:06.070547 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:06.070732 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.070740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.072142 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.072146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.073289 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.073300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.074592 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.074596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.074699 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.074721 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:19:06.074793 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:19:06.074859 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6882,"mutex_wait_us":37,"thread_start_us":91,"threads_started":1}
I20260927 15:19:06.075512 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.063081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.9.242.131:45443)
0927 15:19:06.063803 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:06.063809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:06.069925 (+ 6116us) server_negotiation.cc:407] Connection header received
0927 15:19:06.070162 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:06.070168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:06.070394 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:06.070540 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:06.070879 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.072006 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.073434 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.074282 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.074388 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.074529 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:06.075272 (+ 743us) server_negotiation.cc:299] Negotiation successful
0927 15:19:06.075374 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":62,"threads_started":1}
I20260927 15:19:06.075415 23238 tablet_service.cc:1467] Processing CreateTablet for tablet e4cfddeaa36b4affb0517d0b1dcb7978 (DEFAULT_TABLE table=test-table [id=ba909a379c874e5797c2c4a8c0e50d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:06.076617 23238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cfddeaa36b4affb0517d0b1dcb7978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:06.078270 23371 tablet_service.cc:1467] Processing CreateTablet for tablet e4cfddeaa36b4affb0517d0b1dcb7978 (DEFAULT_TABLE table=test-table [id=ba909a379c874e5797c2c4a8c0e50d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:06.079255 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cfddeaa36b4affb0517d0b1dcb7978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:06.081065 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.062862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:46117 (local address 127.0.0.1:39792)
0927 15:19:06.070205 (+ 7343us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:06.070218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:06.070301 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:06.076395 (+ 6094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:06.076398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:06.076416 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:06.076494 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.076500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.077950 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.077953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.079233 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.079244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.080682 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.080686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.080772 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.080793 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:19:06.080859 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:19:06.080925 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7284,"mutex_wait_us":425,"thread_start_us":79,"threads_started":1}
I20260927 15:19:06.081574 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.063040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.9.242.129:46117)
0927 15:19:06.069690 (+ 6650us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:06.069697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:06.075547 (+ 5850us) server_negotiation.cc:407] Connection header received
0927 15:19:06.075639 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:06.075644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:06.075852 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:06.076242 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:06.076612 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.077826 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.079370 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.080399 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.080492 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.080621 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:06.081342 (+ 721us) server_negotiation.cc:299] Negotiation successful
0927 15:19:06.081443 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6504,"thread_start_us":112,"threads_started":1}
I20260927 15:19:06.084543 23104 tablet_service.cc:1467] Processing CreateTablet for tablet e4cfddeaa36b4affb0517d0b1dcb7978 (DEFAULT_TABLE table=test-table [id=ba909a379c874e5797c2c4a8c0e50d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:06.085675 23104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4cfddeaa36b4affb0517d0b1dcb7978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:06.088650 23472 tablet_bootstrap.cc:492] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Bootstrap starting.
I20260927 15:19:06.091074 23472 tablet_bootstrap.cc:654] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:06.091997 23472 log.cc:824] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:06.092739 23473 tablet_bootstrap.cc:492] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Bootstrap starting.
I20260927 15:19:06.095825 23472 tablet_bootstrap.cc:492] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: No bootstrap required, opened a new log
I20260927 15:19:06.096040 23472 ts_tablet_manager.cc:1397] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 15:19:06.099278 23475 tablet_bootstrap.cc:492] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f: Bootstrap starting.
I20260927 15:19:06.100970 23473 tablet_bootstrap.cc:654] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:06.100775 23472 raft_consensus.cc:348] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.101068 23472 raft_consensus.cc:374] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:06.101123 23472 raft_consensus.cc:729] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43b753ef32c459a878c43577a567ea8, State: Initialized, Role: FOLLOWER
I20260927 15:19:06.101588 23475 tablet_bootstrap.cc:654] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:06.101613 23472 consensus_queue.cc:260] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.101854 23473 log.cc:824] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:06.102543 23475 log.cc:824] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:06.102823 23472 ts_tablet_manager.cc:1428] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 15:19:06.103194 23304 heartbeater.cc:503] Master 127.9.242.190:45571 was elected leader, sending a full tablet report...
I20260927 15:19:06.105612 23473 tablet_bootstrap.cc:492] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: No bootstrap required, opened a new log
I20260927 15:19:06.105846 23473 ts_tablet_manager.cc:1397] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20260927 15:19:06.109068 23473 raft_consensus.cc:348] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.109277 23473 raft_consensus.cc:374] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:06.109308 23473 raft_consensus.cc:729] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac819aa12914c6ab7e6bdea45b9a5ca, State: Initialized, Role: FOLLOWER
I20260927 15:19:06.109732 23473 consensus_queue.cc:260] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.110417 23438 heartbeater.cc:503] Master 127.9.242.190:45571 was elected leader, sending a full tablet report...
I20260927 15:19:06.110642 23473 ts_tablet_manager.cc:1428] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 15:19:06.113937 23475 tablet_bootstrap.cc:492] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f: No bootstrap required, opened a new log
I20260927 15:19:06.114154 23475 ts_tablet_manager.cc:1397] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260927 15:19:06.114900 23479 raft_consensus.cc:482] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:06.115044 23479 raft_consensus.cc:504] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.117686 23479 leader_election.cc:288] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c43b753ef32c459a878c43577a567ea8 (127.9.242.130:34615), cf08acf68b2d4c2b94522bd090de3b6f (127.9.242.129:46117)
I20260927 15:19:06.118175 23475 raft_consensus.cc:348] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.118414 23475 raft_consensus.cc:374] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:06.118464 23475 raft_consensus.cc:729] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf08acf68b2d4c2b94522bd090de3b6f, State: Initialized, Role: FOLLOWER
I20260927 15:19:06.118865 23475 consensus_queue.cc:260] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.119445 23170 heartbeater.cc:503] Master 127.9.242.190:45571 was elected leader, sending a full tablet report...
I20260927 15:19:06.119853 23475 ts_tablet_manager.cc:1428] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:19:06.124759 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.116810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:48949 (local address 127.9.242.130:34615)
0927 15:19:06.116922 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:06.116926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:06.120034 (+ 3108us) server_negotiation.cc:407] Connection header received
0927 15:19:06.120578 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:06.120581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:06.120630 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:06.120673 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:06.121086 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.121893 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.123427 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.124035 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.124124 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.124176 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:06.124557 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 15:19:06.124633 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:19:06.125177 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.116470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:34615 (local address 127.9.242.131:48949)
0927 15:19:06.119945 (+ 3475us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:06.119960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:06.120466 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:06.120811 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:06.120814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:06.120830 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:06.120965 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.120972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.122026 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.122030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.123282 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.123305 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.124315 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.124319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.124417 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.124441 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:06.124943 (+ 502us) client_negotiation.cc:240] Negotiation successful
0927 15:19:06.125018 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3405,"thread_start_us":97,"threads_started":1}
I20260927 15:19:06.125828 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cfddeaa36b4affb0517d0b1dcb7978" candidate_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43b753ef32c459a878c43577a567ea8" is_pre_election: true
I20260927 15:19:06.126135 23258 raft_consensus.cc:2454] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac819aa12914c6ab7e6bdea45b9a5ca in term 0.
I20260927 15:19:06.126654 23325 leader_election.cc:302] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac819aa12914c6ab7e6bdea45b9a5ca, c43b753ef32c459a878c43577a567ea8; no voters:
I20260927 15:19:06.126963 23479 raft_consensus.cc:2781] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:06.127045 23479 raft_consensus.cc:482] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:06.127089 23479 raft_consensus.cc:3037] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:06.127369 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.117538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:60017 (local address 127.9.242.129:46117)
0927 15:19:06.117639 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:06.117643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:06.120320 (+ 2677us) server_negotiation.cc:407] Connection header received
0927 15:19:06.121246 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:06.121249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:06.121292 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:06.121363 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:06.123442 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.124230 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.125847 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.126446 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.126534 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.126584 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:06.127175 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 15:19:06.127255 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:19:06.127897 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.117220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:46117 (local address 127.9.242.131:60017)
0927 15:19:06.120150 (+ 2930us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:06.120160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:06.120246 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:06.123173 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:06.123177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:06.123197 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:06.123315 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.123321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.124398 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.124402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.125712 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:06.125723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.126824 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.126830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:06.126925 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.126950 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:19:06.127698 (+ 748us) client_negotiation.cc:240] Negotiation successful
0927 15:19:06.127762 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2880,"thread_start_us":91,"threads_started":1}
I20260927 15:19:06.128430 23479 raft_consensus.cc:504] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.128513 23123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cfddeaa36b4affb0517d0b1dcb7978" candidate_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" is_pre_election: true
I20260927 15:19:06.128791 23479 leader_election.cc:288] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [CANDIDATE]: Term 1 election: Requested vote from peers c43b753ef32c459a878c43577a567ea8 (127.9.242.130:34615), cf08acf68b2d4c2b94522bd090de3b6f (127.9.242.129:46117)
I20260927 15:19:06.128786 23123 raft_consensus.cc:2454] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ac819aa12914c6ab7e6bdea45b9a5ca in term 0.
I20260927 15:19:06.129227 23123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cfddeaa36b4affb0517d0b1dcb7978" candidate_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf08acf68b2d4c2b94522bd090de3b6f"
I20260927 15:19:06.129336 23123 raft_consensus.cc:3037] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:06.129546 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4cfddeaa36b4affb0517d0b1dcb7978" candidate_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c43b753ef32c459a878c43577a567ea8"
I20260927 15:19:06.129693 23258 raft_consensus.cc:3037] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:06.130586 23123 raft_consensus.cc:2454] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac819aa12914c6ab7e6bdea45b9a5ca in term 1.
I20260927 15:19:06.130918 23258 raft_consensus.cc:2454] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ac819aa12914c6ab7e6bdea45b9a5ca in term 1.
I20260927 15:19:06.130913 23327 leader_election.cc:302] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac819aa12914c6ab7e6bdea45b9a5ca, cf08acf68b2d4c2b94522bd090de3b6f; no voters:
I20260927 15:19:06.131207 23479 raft_consensus.cc:2781] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:06.131454 23479 raft_consensus.cc:686] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 1 LEADER]: Becoming Leader. State: Replica: 1ac819aa12914c6ab7e6bdea45b9a5ca, State: Running, Role: LEADER
I20260927 15:19:06.131801 23479 consensus_queue.cc:237] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } }
I20260927 15:19:06.135313 22966 catalog_manager.cc:5495] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ac819aa12914c6ab7e6bdea45b9a5ca (127.9.242.131). New cstate: current_term: 1 leader_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac819aa12914c6ab7e6bdea45b9a5ca" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:06.159631 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.154578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.9.242.131:45443)
0927 15:19:06.154703 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:06.154708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:06.154721 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:19:06.154801 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:06.154804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:06.154855 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:06.154944 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:06.155642 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.156406 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.157585 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:06.157930 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:06.158021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:06.158425 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:06.158981 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:06.159094 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:06.159414 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 15:19:06.159473 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:19:06.169731 23123 raft_consensus.cc:1261] T e4cfddeaa36b4affb0517d0b1dcb7978 P cf08acf68b2d4c2b94522bd090de3b6f [term 1 FOLLOWER]: Refusing update from remote peer 1ac819aa12914c6ab7e6bdea45b9a5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:06.171020 23258 raft_consensus.cc:1261] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 1 FOLLOWER]: Refusing update from remote peer 1ac819aa12914c6ab7e6bdea45b9a5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:06.170976 23479 consensus_queue.cc:1035] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf08acf68b2d4c2b94522bd090de3b6f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:06.175187 23479 consensus_queue.cc:1035] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "c43b753ef32c459a878c43577a567ea8" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 34615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:06.200512 23487 mvcc.cc:205] Tried to move back new op lower bound from 7333979529902735360 to 7333979529758941184. Current Snapshot: MvccSnapshot[applied={T|T < 7333979529902735360 or (T in {7333979529902735360})}]
I20260927 15:19:06.207386 22966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:06.207595 22966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:06.210517 22966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fe27249a8044fc6b302dc605d9b5427: Sending DeleteTablet for 3 replicas of tablet e4cfddeaa36b4affb0517d0b1dcb7978
I20260927 15:19:06.211344 23238 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cfddeaa36b4affb0517d0b1dcb7978 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:06 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:36118
I20260927 15:19:06.212371 23371 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cfddeaa36b4affb0517d0b1dcb7978 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:06 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:53014
I20260927 15:19:06.213280 23104 tablet_service.cc:1514] Processing DeleteTablet for tablet e4cfddeaa36b4affb0517d0b1dcb7978 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:06 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:39792
I20260927 15:19:06.215278 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23010
I20260927 15:19:06.216001 23496 tablet_replica.cc:330] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: stopping tablet replica
I20260927 15:19:06.216359 23496 raft_consensus.cc:2229] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:06.216709 23496 raft_consensus.cc:2258] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:06.217052 23497 tablet_replica.cc:330] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: stopping tablet replica
I20260927 15:19:06.217350 23497 raft_consensus.cc:2229] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:06.222823 23496 ts_tablet_manager.cc:1905] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:06.224093 23497 raft_consensus.cc:2258] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:06.227205 23497 ts_tablet_manager.cc:1905] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:06.229700 23496 ts_tablet_manager.cc:1918] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:06.229818 23496 log.cc:1196] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/e4cfddeaa36b4affb0517d0b1dcb7978
I20260927 15:19:06.230151 23496 ts_tablet_manager.cc:1939] T e4cfddeaa36b4affb0517d0b1dcb7978 P c43b753ef32c459a878c43577a567ea8: Deleting consensus metadata
I20260927 15:19:06.235903 23497 ts_tablet_manager.cc:1918] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:06.236021 23497 log.cc:1196] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/e4cfddeaa36b4affb0517d0b1dcb7978
I20260927 15:19:06.236353 23497 ts_tablet_manager.cc:1939] T e4cfddeaa36b4affb0517d0b1dcb7978 P 1ac819aa12914c6ab7e6bdea45b9a5ca: Deleting consensus metadata
W20260927 15:19:06.238099 22958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:46117 (error 108)
I20260927 15:19:06.239707 22956 catalog_manager.cc:4841] TS c43b753ef32c459a878c43577a567ea8 (127.9.242.130:34615): tablet e4cfddeaa36b4affb0517d0b1dcb7978 (table test-table [id=ba909a379c874e5797c2c4a8c0e50d75]) successfully deleted
I20260927 15:19:06.240026 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23174
I20260927 15:19:06.246822 22956 catalog_manager.cc:4841] TS 1ac819aa12914c6ab7e6bdea45b9a5ca (127.9.242.131:45443): tablet e4cfddeaa36b4affb0517d0b1dcb7978 (table test-table [id=ba909a379c874e5797c2c4a8c0e50d75]) successfully deleted
I20260927 15:19:06.251878 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:06.251520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:46117 (local address 127.0.0.1:39806)
0927 15:19:06.251703 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:06.251776 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260927 15:19:06.252070 22958 catalog_manager.cc:4584] TS cf08acf68b2d4c2b94522bd090de3b6f (127.9.242.129:46117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4cfddeaa36b4affb0517d0b1dcb7978: Network error: Client connection negotiation failed: client connection to 127.9.242.129:46117: connect: Connection refused (error 111)
I20260927 15:19:06.253496 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23306
I20260927 15:19:06.261605 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 22939
I20260927 15:19:06.268961 10186 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 15:19:06.613749 10186 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 15:19:06.613860 10186 mini_ranger.cc:66] Stopping Ranger...
I20260927 15:19:06.948505 10186 mini_ranger.cc:68] Stopped Ranger
2026-09-27 15:19:06.948 UTC [22006] LOG: received smart shutdown request
2026-09-27 15:19:06.952 UTC [22006] LOG: background worker "logical replication launcher" (PID 22015) exited with exit code 1
2026-09-27 15:19:06.952 UTC [22010] LOG: shutting down
2026-09-27 15:19:07.120 UTC [22006] LOG: database system is shut down
2026-09-27T15:19:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47903 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:07 dist-test-slave-m4lt krb5kdc[23508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:07 dist-test-slave-m4lt krb5kdc[23508](info): set up 2 sockets
Sep 27 15:19:07 dist-test-slave-m4lt krb5kdc[23508](info): commencing operation
krb5kdc: starting...
W20260927 15:19:08.173465 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:19:08.293605 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:42041
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:42041,127.9.242.189:36215,127.9.242.188:36391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:08.401727 23521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:08.401999 23521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:08.402055 23521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:08.406045 23521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:08.406107 23521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:08.406142 23521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:08.406174 23521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:08.406203 23521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:08.411514 23521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:42041,127.9.242.189:36215,127.9.242.188:36391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:08.412529 23521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:08.413523 23521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:08.421494 23526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:08.421494 23527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:08.421509 23529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:08.421844 23521 server_base.cc:1034] running on GCE node
I20260927 15:19:08.422444 23521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:08.423533 23521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:08.424724 23521 hybrid_clock.cc:648] HybridClock initialized: now 1790522348424706 us; error 30 us; skew 500 ppm
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:08.427378 23521 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:08.428200 23521 webserver.cc:460] Webserver started at http://127.9.242.190:41067/ using document root <none> and password file <none>
I20260927 15:19:08.428773 23521 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:08.428839 23521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:08.429026 23521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:08.431007 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "c52a62c865214ff9ae05055a5ece2c57"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "669e422475bc1ba04987a624693dd161"
server_key_iv: "1ef0d95fa9e5d7020c32b60681fb0b0c"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.431528 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "c52a62c865214ff9ae05055a5ece2c57"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "669e422475bc1ba04987a624693dd161"
server_key_iv: "1ef0d95fa9e5d7020c32b60681fb0b0c"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.435170 23521 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 15:19:08.437541 23536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:08.438788 23521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:19:08.438884 23521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "c52a62c865214ff9ae05055a5ece2c57"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "669e422475bc1ba04987a624693dd161"
server_key_iv: "1ef0d95fa9e5d7020c32b60681fb0b0c"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.438971 23521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:08.451421 23521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:08.454119 23521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:08.454272 23521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:08.460477 23521 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:42041
I20260927 15:19:08.460528 23588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:42041 every 8 connection(s)
I20260927 15:19:08.461380 23521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:08.464143 23589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:08.467916 23589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:19:08.469223 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23521
I20260927 15:19:08.469332 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:19:08.469597 10186 external_mini_cluster.cc:1428] Setting key 4cb4680e5f96318a63ad8c0e4317fb4b
WARNING: no policy specified for kudu/127.9.242.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.189@KRBTEST.COM" created.
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:08.486407 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.470254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42041 (local address 127.0.0.1:48038)
0927 15:19:08.470789 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.470825 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.471658 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.472306 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.472311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.472832 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.473071 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.473087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.474765 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.474768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.475491 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.475501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.476126 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.476130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.476238 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.476757 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.476773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.480630 (+ 3857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.483105 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.483116 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.483127 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.483445 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.483845 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.483849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.483851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.484023 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.484887 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.484895 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.485038 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.485182 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.485472 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"spinlock_wait_cycles":26368,"thread_start_us":157,"threads_started":1}
I20260927 15:19:08.486415 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.470253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.9.242.190:42041)
0927 15:19:08.470667 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.470686 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.470922 (+ 236us) server_negotiation.cc:407] Connection header received
0927 15:19:08.471943 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.471947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.472122 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.472293 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.473261 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.474645 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.475598 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.475983 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.476163 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.480835 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.480840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.480851 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.480873 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.482951 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.483546 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.483551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.483553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.483727 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.484131 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.484135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.484137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.484691 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.484840 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.485305 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.485593 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260927 15:19:08.488783 23589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:19:08.489486 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.489259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43896)
0927 15:19:08.489363 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.489431 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:19:08.489734 23539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
I20260927 15:19:08.490499 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.490297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43900)
0927 15:19:08.490396 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.490443 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:19:08.490618 23589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:36215: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab.
Entry for principal kudu/127.9.242.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab.
I20260927 15:19:08.496359 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:36215
--webserver_interface=127.9.242.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab
--principal=kudu/127.9.242.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:42041,127.9.242.189:36215,127.9.242.188:36391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:08.539870 23589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } attempt: 1
I20260927 15:19:08.540808 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.540480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43910)
0927 15:19:08.540640 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.540715 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:19:08.541501 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.541291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43924)
0927 15:19:08.541387 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.541435 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 15:19:08.541618 23589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:36215: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
W20260927 15:19:08.603041 23596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:08.603292 23596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:08.603331 23596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:08.607259 23596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:08.607311 23596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:08.607329 23596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:08.607344 23596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:08.607357 23596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:08.609781 23589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } attempt: 2
I20260927 15:19:08.610553 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.610249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43930)
0927 15:19:08.610401 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.610469 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:19:08.611207 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.611006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43936)
0927 15:19:08.611093 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.611140 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:19:08.611318 23589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:36215: Network error: Client connection negotiation failed: client connection to 127.9.242.189:36215: connect: Connection refused (error 111)
I20260927 15:19:08.612835 23596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:42041,127.9.242.189:36215,127.9.242.188:36391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab
--principal=kudu/127.9.242.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:36215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.242.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:08.613830 23596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:08.614777 23596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:08.622910 23604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:08.622987 23602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:08.623093 23596 server_base.cc:1034] running on GCE node
W20260927 15:19:08.622910 23601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:08.623740 23596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:08.624811 23596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:08.625984 23596 hybrid_clock.cc:648] HybridClock initialized: now 1790522348625955 us; error 39 us; skew 500 ppm
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:08.628700 23596 init.cc:377] Logged in from keytab as kudu/127.9.242.189@KRBTEST.COM (short username kudu)
I20260927 15:19:08.629528 23596 webserver.cc:460] Webserver started at http://127.9.242.189:44863/ using document root <none> and password file <none>
I20260927 15:19:08.630067 23596 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:08.630127 23596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:08.630292 23596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:08.632288 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data/instance:
uuid: "3fe0ac2f88934db0a5ea655de84da25d"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "b8a346a597654a9fb7c49f42dddd401f"
server_key_iv: "9822c41ac6a6a680ddd934956a411755"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.632810 23596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/wal/instance:
uuid: "3fe0ac2f88934db0a5ea655de84da25d"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "b8a346a597654a9fb7c49f42dddd401f"
server_key_iv: "9822c41ac6a6a680ddd934956a411755"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.636405 23596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 15:19:08.638995 23611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:08.640268 23596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:19:08.640365 23596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/wal
uuid: "3fe0ac2f88934db0a5ea655de84da25d"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "b8a346a597654a9fb7c49f42dddd401f"
server_key_iv: "9822c41ac6a6a680ddd934956a411755"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.640451 23596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:08.653640 23596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:08.656711 23596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:08.656859 23596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:08.663514 23596 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.189:36215
I20260927 15:19:08.663619 23663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.189:36215 every 8 connection(s)
I20260927 15:19:08.664462 23596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
I20260927 15:19:08.667212 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:08.670863 23664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:19:08.672089 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23596
I20260927 15:19:08.672192 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-1/wal/instance
I20260927 15:19:08.672459 10186 external_mini_cluster.cc:1428] Setting key 92896c8fbd4f60b59deeb568f7f76a35
WARNING: no policy specified for kudu/127.9.242.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.188@KRBTEST.COM" created.
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:08.687623 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.672768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.9.242.190:42041)
0927 15:19:08.672886 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.672890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.673893 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 15:19:08.675045 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.675048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.675098 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.675176 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.676397 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.676941 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.678103 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.678345 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.678436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.683913 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.683917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.683919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.683947 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.685504 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.686059 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.686062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.686063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.686174 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.686516 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.686519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.686520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.686724 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.686811 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.687388 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.687498 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:19:08.688797 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.673252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42041 (local address 127.0.0.1:48054)
0927 15:19:08.673744 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.673787 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.674810 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.675347 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.675356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.675692 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.676253 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.676266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.677063 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.677066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.677990 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.677999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.678484 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.678488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.678648 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.679265 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.679281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.683549 (+ 4268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.685633 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.685638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.685670 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.685945 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.686266 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.686268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.686270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.686418 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.686829 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.686836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.687106 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.687319 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.687700 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260927 15:19:08.690292 23664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
I20260927 15:19:08.699206 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.690819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43938)
0927 15:19:08.690981 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.690994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.691067 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.691560 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.691563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.691709 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.691829 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.691836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.692903 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.692906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.693602 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.693612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.694291 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.694294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.694385 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.694826 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.694837 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.696045 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.697858 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.697861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.697863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.698028 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.698266 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.698268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.698270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.698338 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.698738 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.698741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.698820 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.698910 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.699026 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 15:19:08.699245 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.690894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.9.242.189:36215)
0927 15:19:08.691198 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.691215 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.691237 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:19:08.691271 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.691275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.691428 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.691533 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.691932 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.692767 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.693767 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.694170 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.694250 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.696191 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.696199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.696209 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.696228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.697759 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.698118 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.698124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.698125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.698191 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.698409 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.698412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.698414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.698647 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.698758 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.698994 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.699111 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab.
Entry for principal kudu/127.9.242.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab.
I20260927 15:19:08.700855 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:36391
--webserver_interface=127.9.242.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab
--principal=kudu/127.9.242.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:39961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:42041,127.9.242.189:36215,127.9.242.188:36391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:08.701398 23664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:19:08.702164 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.701884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59832)
0927 15:19:08.702004 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.702087 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:19:08.702342 23613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
I20260927 15:19:08.702455 23589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } attempt: 3
I20260927 15:19:08.703435 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.703244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59848)
0927 15:19:08.703338 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.703378 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 15:19:08.703560 23664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:36391: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
I20260927 15:19:08.711411 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.702846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43942)
0927 15:19:08.702965 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.702978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.703053 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.703378 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.703381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.703539 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.703690 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.703696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.704724 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.704728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.705771 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.705781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.706384 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.706387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.706505 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.706987 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.707000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.708345 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.710096 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.710099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.710101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.710289 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.710560 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.710562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.710564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.710642 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.711026 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.711028 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.711115 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.711193 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.711292 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:19:08.711664 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.703067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.9.242.189:36215)
0927 15:19:08.703162 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.703166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.703177 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:19:08.703208 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.703211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.703249 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.703793 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.703872 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.704579 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.705878 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.706252 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.706360 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.708480 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.708484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.708487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.708513 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.709979 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.710380 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.710383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.710385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.710470 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.710720 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.710723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.710724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.710934 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.711034 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.711263 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.711371 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 15:19:08.712682 23589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:19:08.713326 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.713082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59860)
0927 15:19:08.713200 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.713255 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 15:19:08.714008 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.713775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59872)
0927 15:19:08.713904 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.713951 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 15:19:08.714133 23589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:36391: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
I20260927 15:19:08.745306 23589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } attempt: 1
I20260927 15:19:08.746186 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.745856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59876)
0927 15:19:08.746018 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.746090 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:19:08.746933 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.746719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59878)
0927 15:19:08.746820 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.746873 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:19:08.747083 23589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:36391: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
I20260927 15:19:08.752821 23664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } attempt: 1
I20260927 15:19:08.753582 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.753309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59882)
0927 15:19:08.753448 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.753510 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 15:19:08.754400 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.754172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59884)
0927 15:19:08.754299 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.754345 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 15:19:08.754526 23664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:36391: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
W20260927 15:19:08.812238 23670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:08.812547 23670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:08.812608 23670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:08.816728 23670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:08.816797 23670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:08.816823 23670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:08.816844 23670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:08.816866 23670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:08.822235 23670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:39961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:42041,127.9.242.189:36215,127.9.242.188:36391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab
--principal=kudu/127.9.242.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:36391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.242.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:08.822255 23589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } attempt: 2
I20260927 15:19:08.822674 23664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } attempt: 2
I20260927 15:19:08.823060 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.822707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59894)
0927 15:19:08.822897 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.822970 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:19:08.823268 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.823027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59904)
0927 15:19:08.823147 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.823198 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:19:08.823542 23670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:08.823882 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.823614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59912)
0927 15:19:08.823763 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.823811 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 15:19:08.824002 23589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:36391: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
I20260927 15:19:08.824014 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.823797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59924)
0927 15:19:08.823899 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.823947 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 15:19:08.824136 23664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:36391: Network error: Client connection negotiation failed: client connection to 127.9.242.188:36391: connect: Connection refused (error 111)
I20260927 15:19:08.824776 23670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:08.833114 23677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:08.833114 23679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:08.833114 23676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:08.833351 23670 server_base.cc:1034] running on GCE node
I20260927 15:19:08.834075 23670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:08.835196 23670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:08.836374 23670 hybrid_clock.cc:648] HybridClock initialized: now 1790522348836354 us; error 31 us; skew 500 ppm
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:08.839188 23670 init.cc:377] Logged in from keytab as kudu/127.9.242.188@KRBTEST.COM (short username kudu)
I20260927 15:19:08.840111 23670 webserver.cc:460] Webserver started at http://127.9.242.188:37405/ using document root <none> and password file <none>
I20260927 15:19:08.840642 23670 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:08.840723 23670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:08.841025 23670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:08.843339 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data/instance:
uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "8f60e83561616aa7978c361096dcb74f"
server_key_iv: "ec35ff4d215546e79fbd3a38562f8d7b"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.843928 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/wal/instance:
uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "8f60e83561616aa7978c361096dcb74f"
server_key_iv: "ec35ff4d215546e79fbd3a38562f8d7b"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.847694 23670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:19:08.850185 23686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:08.851418 23670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:08.851548 23670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/wal
uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b"
format_stamp: "Formatted at 2026-09-27 15:19:08 on dist-test-slave-m4lt"
server_key: "8f60e83561616aa7978c361096dcb74f"
server_key_iv: "ec35ff4d215546e79fbd3a38562f8d7b"
server_key_version: "encryptionkey@0"
I20260927 15:19:08.851652 23670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:08.876860 23670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:08.879696 23670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:08.879887 23670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:08.886557 23670 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.188:36391
I20260927 15:19:08.886658 23738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.188:36391 every 8 connection(s)
I20260927 15:19:08.887512 23670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
I20260927 15:19:08.890394 23739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:08.894124 23739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:19:08.897786 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23670
I20260927 15:19:08.897894 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790522194092531-10186-0/minicluster-data/master-2/wal/instance
I20260927 15:19:08.898173 10186 external_mini_cluster.cc:1428] Setting key a54ac21f4b4b408dbda61c3abcf69d65
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:19:08.907377 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.895933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.9.242.190:42041)
0927 15:19:08.896073 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.896077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.896865 (+ 788us) server_negotiation.cc:407] Connection header received
0927 15:19:08.897784 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.897787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.897855 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.897937 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.899367 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.899983 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.901092 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.901308 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.901383 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.903926 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.903929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.903931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.903955 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.905365 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.905855 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.905859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.905860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.905932 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.906234 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.906237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.906239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.906452 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.906552 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.907147 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.907254 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:19:08.908416 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.896280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:42041 (local address 127.0.0.1:48068)
0927 15:19:08.896709 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.896759 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.897545 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.898184 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.898193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.898596 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.899223 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.899243 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.900105 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.900108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.900968 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.900977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.901440 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.901444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.901637 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.902241 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.902256 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.903800 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.905462 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.905470 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.905480 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.905767 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.906010 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.906013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.906015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.906145 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.906579 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.906586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.906898 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.907094 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.907410 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":6,"thread_start_us":112,"threads_started":1}
I20260927 15:19:08.908419 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.898800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.9.242.188:36391)
0927 15:19:08.899105 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.899116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.899136 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:19:08.899253 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.899257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.899430 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.899586 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.900205 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.900988 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.901966 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.902309 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.902398 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.904854 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.904861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.904874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.904898 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.906287 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.906728 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.906734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.906736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.906826 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.907066 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.907069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.907071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.907338 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.907519 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.907821 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.907952 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":221,"thread_start_us":125,"threads_started":1}
I20260927 15:19:08.909768 23739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:08.915279 23664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } attempt: 3
I20260927 15:19:08.918565 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.910420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.9.242.189:36215)
0927 15:19:08.910539 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.910544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.910600 (+ 56us) server_negotiation.cc:407] Connection header received
0927 15:19:08.910765 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.910769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.910821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.910892 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.911316 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.911883 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.912682 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.912979 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.913050 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.915272 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.915277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.915280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.915306 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.916911 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.917293 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.917297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.917299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.917384 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.917612 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.917615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.917617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.917870 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.917955 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.918285 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.918401 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:19:08.919503 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.910264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:36215 (local address 127.0.0.1:43948)
0927 15:19:08.910537 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.910550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.910664 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.910913 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.910916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.911092 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.911209 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.911214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.911981 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.911984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.912578 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.912586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.913065 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.913069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.913188 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.913620 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.913634 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.915077 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.917014 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.917017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.917019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.917202 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.917471 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.917473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.917474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.917539 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.917998 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.918001 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.918096 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.919209 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.919350 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260927 15:19:08.919611 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.910648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.9.242.190:42041)
0927 15:19:08.910756 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.910759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.910772 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:19:08.910854 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.910856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.910894 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.910965 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.911525 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.912093 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.912870 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.913086 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.913179 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.916082 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.916086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.916088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.916114 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.917739 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.918205 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.918209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.918211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.918291 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.918574 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.918577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.918579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.918818 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.919010 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.919363 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.919473 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:19:08.920547 23739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:19:08.925423 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.915679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59938)
0927 15:19:08.915866 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.915880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.915963 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.916492 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.916495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.916772 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.916881 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.916889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.918625 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.918629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.919464 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.919475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.920130 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.920133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.920214 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.920690 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.920703 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.922124 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.923814 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.923817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.923819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.924048 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.924358 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.924361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.924363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.924454 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.924911 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.924914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.925015 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.925113 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.925257 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 15:19:08.925890 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.915752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.9.242.188:36391)
0927 15:19:08.915874 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.915878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.915922 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:19:08.916323 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.916325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.916362 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.916470 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.917024 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.918494 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.919607 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.919997 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.920128 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.922256 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.922259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.922262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.922285 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.923687 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.924163 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.924166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.924168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.924244 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.924562 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.924565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.924566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.924786 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.924915 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.925617 (+ 702us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.925750 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:19:08.926199 23589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } attempt: 3
I20260927 15:19:08.929725 23664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: Bootstrap starting.
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
I20260927 15:19:08.933024 23664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:08.933194 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.921054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59940)
0927 15:19:08.921464 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.921477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.921547 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.921832 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.921835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.921990 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.922076 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.922081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.923598 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.923601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.924280 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.924290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.924820 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.924823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.924894 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.925306 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.925319 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.926988 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.931776 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.931779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.931781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.931990 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.932239 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.932242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.932243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.932315 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.932750 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.932752 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.932829 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.932916 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.933035 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
I20260927 15:19:08.933786 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.921192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.9.242.188:36391)
0927 15:19:08.921530 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.921539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.921554 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:19:08.921633 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.921636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.921723 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.921805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.922201 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.923486 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.924387 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.924727 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.924808 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.927112 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.927116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.927118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.927143 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.931672 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.932075 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.932077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.932079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.932159 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.932393 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.932395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.932397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.932635 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.932718 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.933440 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.933542 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":148,"threads_started":1}
I20260927 15:19:08.933982 23664 log.cc:824] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:19:08 dist-test-slave-m4lt krb5kdc[23508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522348, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:19:08.936702 23739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: Bootstrap starting.
I20260927 15:19:08.936904 23664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: No bootstrap required, opened a new log
I20260927 15:19:08.937922 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.926343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.9.242.189:36215)
0927 15:19:08.926453 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.926457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.926470 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:19:08.926503 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.926506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.926549 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.926620 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.928093 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.928567 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.929543 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.929943 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.930032 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.933511 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.933515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.933517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.933541 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.935115 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.935569 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.935573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.935574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.935671 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.935974 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.935978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.935980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.936442 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.937543 (+ 1101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.937620 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.937778 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:19:08.939145 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.927545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:36391 (local address 127.0.0.1:59946)
0927 15:19:08.927642 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:08.927655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:08.927723 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:08.930156 (+ 2433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:08.930160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:08.930338 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:08.930443 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.930450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.931253 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.931257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.932072 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:08.932083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.932673 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.932677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:08.932769 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.933797 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:08.933811 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:08.935804 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:08.937537 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.937541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.937542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.937826 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.938154 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:08.938158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:08.938159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:08.938252 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:08.938688 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:08.938691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:08.938787 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 15:19:08.938865 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:19:08.938996 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 15:19:08.939618 23739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:08.940423 23739 log.cc:824] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:08.940958 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.929783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.9.242.188:36391)
0927 15:19:08.929924 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.929928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.929941 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:19:08.929969 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.929972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.930011 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.930114 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.930571 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.931145 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.932229 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.932542 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.934976 (+ 2434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.935943 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.935946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.935949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.935972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.937417 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.937944 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.937947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.937949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.938038 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.938377 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.938379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.938381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.938585 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.939425 (+ 840us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.939494 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.939607 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:19:08.941321 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:19:08.941798 23664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
I20260927 15:19:08.942018 23664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:08.942062 23664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe0ac2f88934db0a5ea655de84da25d, State: Initialized, Role: FOLLOWER
I20260927 15:19:08.942524 23664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
I20260927 15:19:08.942703 23739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: No bootstrap required, opened a new log
I20260927 15:19:08.943953 23664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:08.944044 23589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57: Bootstrap starting.
I20260927 15:19:08.945820 23747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:08.945951 23747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:19:08.945969 23739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
I20260927 15:19:08.946177 23739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:08.946224 23739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2bc1df15ab46a3b44b2ee1ee00563b, State: Initialized, Role: FOLLOWER
I20260927 15:19:08.946607 23739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
I20260927 15:19:08.946990 23759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:08.947113 23759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:19:08.948930 23589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:08.949693 23589 log.cc:824] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:08.951781 23589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57: No bootstrap required, opened a new log
I20260927 15:19:08.949007 23596 master_runner.cc:186] Error getting master registration for 127.9.242.190:42041: OK, instance_id { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" instance_seqno: 1790522348458674 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:19:08.952368 23596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:19:08.952730 23739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:08.955339 23589 raft_consensus.cc:348] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
I20260927 15:19:08.955569 23589 raft_consensus.cc:374] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:08.955636 23589 raft_consensus.cc:729] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c52a62c865214ff9ae05055a5ece2c57, State: Initialized, Role: FOLLOWER
I20260927 15:19:08.956115 23589 consensus_queue.cc:260] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
W20260927 15:19:08.949018 23758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:19:08.956303 23758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:19:08.957366 23589 sys_catalog.cc:564] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:08.961741 23792 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:08.961881 23792 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:19:08.961988 23792 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:08.962090 23792 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
I20260927 15:19:08.963503 23638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52a62c865214ff9ae05055a5ece2c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe0ac2f88934db0a5ea655de84da25d" is_pre_election: true
I20260927 15:19:08.963829 23638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c52a62c865214ff9ae05055a5ece2c57 in term 0.
I20260927 15:19:08.964187 23539 leader_election.cc:302] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe0ac2f88934db0a5ea655de84da25d, c52a62c865214ff9ae05055a5ece2c57; no voters:
I20260927 15:19:08.957870 23670 master_runner.cc:186] Error getting master registration for 127.9.242.190:42041: OK, instance_id { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" instance_seqno: 1790522348458674 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:19:08.965376 23670 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:19:08.965802 23521 master_runner.cc:263] No leader master found from master c52a62c865214ff9ae05055a5ece2c57
I20260927 15:19:08.965863 23521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 15:19:08.967904 23779 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 15:19:08.967968 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.952550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.9.242.189:36215)
0927 15:19:08.952663 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.952667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.954157 (+ 1490us) server_negotiation.cc:407] Connection header received
0927 15:19:08.954277 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.954280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.954332 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.954430 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.956740 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.957422 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.958661 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.958989 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.959059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.960803 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.960807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.960810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.960835 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.964077 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.966299 (+ 2222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.966303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.966305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.966392 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.966710 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.966713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.966715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.966937 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.967059 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.967676 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.967800 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:19:08.967975 23779 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:19:08.965818 23780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:08.968492 23780 raft_consensus.cc:482] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:08.968549 23780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:08.969527 23713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52a62c865214ff9ae05055a5ece2c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" is_pre_election: true
I20260927 15:19:08.969877 23713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c52a62c865214ff9ae05055a5ece2c57 in term 0.
I20260927 15:19:08.970237 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.950387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.9.242.188:36391)
0927 15:19:08.950494 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.950498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.955686 (+ 5188us) server_negotiation.cc:407] Connection header received
0927 15:19:08.955903 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.955906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.955953 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.955999 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.959146 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.959819 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.961623 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.961944 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.962011 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.966413 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.966417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.966420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.966444 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.967900 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.968720 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.968723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.968725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.968811 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.969094 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.969097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.969098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.969327 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.969449 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.969973 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.970085 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:19:08.971441 23780 raft_consensus.cc:504] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
I20260927 15:19:08.972169 23638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52a62c865214ff9ae05055a5ece2c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe0ac2f88934db0a5ea655de84da25d"
I20260927 15:19:08.972244 23713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c52a62c865214ff9ae05055a5ece2c57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b"
I20260927 15:19:08.972285 23638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:08.972352 23713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:08.973505 23638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c52a62c865214ff9ae05055a5ece2c57 in term 1.
I20260927 15:19:08.973510 23713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c52a62c865214ff9ae05055a5ece2c57 in term 1.
I20260927 15:19:08.973845 23538 leader_election.cc:302] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b2bc1df15ab46a3b44b2ee1ee00563b, c52a62c865214ff9ae05055a5ece2c57; no voters:
I20260927 15:19:08.974080 23792 leader_election.cc:288] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fe0ac2f88934db0a5ea655de84da25d (127.9.242.189:36215), 1b2bc1df15ab46a3b44b2ee1ee00563b (127.9.242.188:36391)
I20260927 15:19:08.974224 23795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:08.974367 23795 raft_consensus.cc:686] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [term 1 LEADER]: Becoming Leader. State: Replica: c52a62c865214ff9ae05055a5ece2c57, State: Running, Role: LEADER
I20260927 15:19:08.974731 23795 consensus_queue.cc:237] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } }
W20260927 15:19:08.975155 23791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:19:08.975224 23791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:19:08.975729 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:08.950798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.9.242.190:42041)
0927 15:19:08.950896 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:08.950900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:08.953953 (+ 3053us) server_negotiation.cc:407] Connection header received
0927 15:19:08.954383 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:08.954386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:08.954427 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:08.954473 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:08.956433 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.957160 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.960794 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:08.961078 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:08.961154 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:08.964433 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:08.964436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:08.964439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:08.964463 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:08.965894 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.974369 (+ 8475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.974372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.974374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.974473 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:08.974800 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:08.974804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:08.974806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:08.975028 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:08.975136 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:08.975446 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 15:19:08.975567 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 15:19:08.976132 23780 leader_election.cc:288] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [CANDIDATE]: Term 1 election: Requested vote from peers 3fe0ac2f88934db0a5ea655de84da25d (127.9.242.189:36215), 1b2bc1df15ab46a3b44b2ee1ee00563b (127.9.242.188:36391)
I20260927 15:19:08.977141 23792 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52a62c865214ff9ae05055a5ece2c57. Latest consensus state: current_term: 1 leader_uuid: "c52a62c865214ff9ae05055a5ece2c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:08.977299 23792 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:08.984601 23797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:08.985623 23797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:09.000551 23713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [term 1 FOLLOWER]: Refusing update from remote peer c52a62c865214ff9ae05055a5ece2c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:09.000551 23638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [term 1 FOLLOWER]: Refusing update from remote peer c52a62c865214ff9ae05055a5ece2c57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:09.001319 23795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:09.002132 23792 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:09.011524 23747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52a62c865214ff9ae05055a5ece2c57. Latest consensus state: current_term: 1 leader_uuid: "c52a62c865214ff9ae05055a5ece2c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:09.011705 23747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:19:09.012176 23759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c52a62c865214ff9ae05055a5ece2c57. Latest consensus state: current_term: 1 leader_uuid: "c52a62c865214ff9ae05055a5ece2c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:09.012305 23759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:19:09.018074 23792 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c52a62c865214ff9ae05055a5ece2c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:09.018239 23792 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:09.019757 23747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c52a62c865214ff9ae05055a5ece2c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:09.019882 23747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:19:09.020354 23798 mvcc.cc:205] Tried to move back new op lower bound from 7333979541497327616 to 7333979541404475392. Current Snapshot: MvccSnapshot[applied={T|T < 7333979541497327616}]
I20260927 15:19:09.020792 23800 mvcc.cc:205] Tried to move back new op lower bound from 7333979541497327616 to 7333979541404475392. Current Snapshot: MvccSnapshot[applied={T|T < 7333979541497327616}]
I20260927 15:19:09.023716 23780 sys_catalog.cc:455] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c52a62c865214ff9ae05055a5ece2c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:09.023850 23780 sys_catalog.cc:458] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:09.024380 23797 catalog_manager.cc:1347] Generated new cluster ID: d97422a48ba64398959294c39a0121b2
I20260927 15:19:09.024442 23797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:09.024957 23759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c52a62c865214ff9ae05055a5ece2c57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c52a62c865214ff9ae05055a5ece2c57" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 42041 } } peers { permanent_uuid: "3fe0ac2f88934db0a5ea655de84da25d" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 36215 } } peers { permanent_uuid: "1b2bc1df15ab46a3b44b2ee1ee00563b" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 36391 } } }
I20260927 15:19:09.025074 23759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:19:09.025720 23799 mvcc.cc:205] Tried to move back new op lower bound from 7333979541497327616 to 7333979541404475392. Current Snapshot: MvccSnapshot[applied={T|T < 7333979541497327616}]
I20260927 15:19:09.048825 23797 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:09.050144 23797 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:09.067790 23797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c52a62c865214ff9ae05055a5ece2c57: Generated new TSK 0
I20260927 15:19:09.068686 23797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:19:09.957759 23758 catalog_manager.cc:1259] Loaded cluster ID: d97422a48ba64398959294c39a0121b2
I20260927 15:19:09.957834 23758 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: loading cluster ID for follower catalog manager: success
I20260927 15:19:09.959611 23758 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: acquiring CA information for follower catalog manager: success
I20260927 15:19:09.960759 23758 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fe0ac2f88934db0a5ea655de84da25d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:19:09.969193 23779 catalog_manager.cc:1259] Loaded cluster ID: d97422a48ba64398959294c39a0121b2
I20260927 15:19:09.969264 23779 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: loading cluster ID for follower catalog manager: success
I20260927 15:19:09.970884 23779 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: acquiring CA information for follower catalog manager: success
I20260927 15:19:09.972004 23779 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b2bc1df15ab46a3b44b2ee1ee00563b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:19:10.134231 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23521
I20260927 15:19:10.139420 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23596
I20260927 15:19:10.144166 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23670
2026-09-27T15:19:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2925 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:10 dist-test-slave-m4lt krb5kdc[23813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:10 dist-test-slave-m4lt krb5kdc[23813](info): set up 2 sockets
Sep 27 15:19:10 dist-test-slave-m4lt krb5kdc[23813](info): commencing operation
krb5kdc: starting...
W20260927 15:19:11.101752 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:19:11 dist-test-slave-m4lt krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:19:11.220847 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43639
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:43639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:11.330659 23826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:11.330953 23826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:11.331008 23826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:11.335287 23826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:11.335353 23826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:11.335378 23826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:11.335400 23826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:11.335429 23826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:11.340876 23826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:43639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:11.342001 23826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:11.342986 23826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:11.351455 23831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:11.351513 23832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:11.351625 23826 server_base.cc:1034] running on GCE node
W20260927 15:19:11.351866 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:11.352398 23826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:11.353618 23826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:11.354826 23826 hybrid_clock.cc:648] HybridClock initialized: now 1790522351354800 us; error 40 us; skew 500 ppm
Sep 27 15:19:11 dist-test-slave-m4lt krb5kdc[23813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522351, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:11.357510 23826 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:11.358379 23826 webserver.cc:460] Webserver started at http://127.9.242.190:42155/ using document root <none> and password file <none>
I20260927 15:19:11.358888 23826 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:11.358943 23826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:11.359097 23826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:11.361028 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "ac66e216134b4e15a9bf33031169f1f7"
format_stamp: "Formatted at 2026-09-27 15:19:11 on dist-test-slave-m4lt"
server_key: "466bac97ddc74cf4babb7fa376c3a509"
server_key_iv: "6997bc727d1f654c9ebbcbabc0544f8c"
server_key_version: "encryptionkey@0"
I20260927 15:19:11.361527 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "ac66e216134b4e15a9bf33031169f1f7"
format_stamp: "Formatted at 2026-09-27 15:19:11 on dist-test-slave-m4lt"
server_key: "466bac97ddc74cf4babb7fa376c3a509"
server_key_iv: "6997bc727d1f654c9ebbcbabc0544f8c"
server_key_version: "encryptionkey@0"
I20260927 15:19:11.365324 23826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 15:19:11.367853 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:11.369266 23826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 15:19:11.369383 23826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "ac66e216134b4e15a9bf33031169f1f7"
format_stamp: "Formatted at 2026-09-27 15:19:11 on dist-test-slave-m4lt"
server_key: "466bac97ddc74cf4babb7fa376c3a509"
server_key_iv: "6997bc727d1f654c9ebbcbabc0544f8c"
server_key_version: "encryptionkey@0"
I20260927 15:19:11.369498 23826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:11.390424 23826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:11.393754 23826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:11.393940 23826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:11.402896 23826 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:43639
I20260927 15:19:11.402945 23893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:43639 every 8 connection(s)
I20260927 15:19:11.404219 23826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:11.407341 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23826
I20260927 15:19:11.407470 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:19:11.407668 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:11.407902 10186 external_mini_cluster.cc:1428] Setting key 6c4186bdf7ed66de909155895ce98f23
I20260927 15:19:11.415462 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7: Bootstrap starting.
I20260927 15:19:11.418325 23894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7: Neither blocks nor log segments found. Creating new log.
Sep 27 15:19:11 dist-test-slave-m4lt krb5kdc[23813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:11.419170 23894 log.cc:824] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:11.421339 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7: No bootstrap required, opened a new log
I20260927 15:19:11.425084 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:11.409624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.9.242.190:43639)
0927 15:19:11.410079 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:11.410098 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:11.410144 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:19:11.411287 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:11.411313 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:11.411773 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:11.412240 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:11.413091 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:11.414686 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:11.415817 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:11.416553 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:11.416715 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:11.419647 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:11.419653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:11.419671 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:11.419706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:11.421626 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:11.422094 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:11.422100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:11.422106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:11.422252 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:11.422578 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:11.422582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:11.422583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:11.423157 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:11.423350 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:11.423690 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:19:11.424016 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260927 15:19:11.425597 23894 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66e216134b4e15a9bf33031169f1f7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43639 } }
I20260927 15:19:11.425928 23894 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:11.425995 23894 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac66e216134b4e15a9bf33031169f1f7, State: Initialized, Role: FOLLOWER
I20260927 15:19:11.426456 23894 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66e216134b4e15a9bf33031169f1f7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43639 } }
I20260927 15:19:11.426627 23894 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:11.426689 23894 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:11.426784 23894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:11.428049 23894 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66e216134b4e15a9bf33031169f1f7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43639 } }
I20260927 15:19:11.428454 23894 leader_election.cc:302] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac66e216134b4e15a9bf33031169f1f7; no voters:
I20260927 15:19:11.428845 23894 leader_election.cc:288] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:11.428864 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:19:11.428972 23899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:11.429162 23899 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [term 1 LEADER]: Becoming Leader. State: Replica: ac66e216134b4e15a9bf33031169f1f7, State: Running, Role: LEADER
I20260927 15:19:11.429486 23899 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66e216134b4e15a9bf33031169f1f7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43639 } }
I20260927 15:19:11.430289 23894 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:11.431102 23900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac66e216134b4e15a9bf33031169f1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66e216134b4e15a9bf33031169f1f7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43639 } } }
I20260927 15:19:11.431523 23900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:11.431424 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac66e216134b4e15a9bf33031169f1f7. Latest consensus state: current_term: 1 leader_uuid: "ac66e216134b4e15a9bf33031169f1f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac66e216134b4e15a9bf33031169f1f7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43639 } } }
I20260927 15:19:11.431828 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:11.431932 23908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:11.435031 23908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:11.441049 23908 catalog_manager.cc:1347] Generated new cluster ID: d29da9d930024850ad035f0c0a509e6f
I20260927 15:19:11.441167 23908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:11.466678 23908 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:11.468052 23908 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:11.478837 23908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac66e216134b4e15a9bf33031169f1f7: Generated new TSK 0
I20260927 15:19:11.479691 23908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:19:11.484351 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23826
2026-09-27T15:19:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1338 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:11 dist-test-slave-m4lt krb5kdc[23921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:11 dist-test-slave-m4lt krb5kdc[23921](info): set up 2 sockets
Sep 27 15:19:11 dist-test-slave-m4lt krb5kdc[23921](info): commencing operation
krb5kdc: starting...
W20260927 15:19:12.437636 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:19:12 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:19:12.556489 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39655
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:39655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:12.663664 23934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:12.663949 23934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:12.663990 23934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:12.668046 23934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:12.668118 23934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:12.668135 23934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:12.668150 23934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:12.668164 23934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:12.673451 23934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:39655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:12.674533 23934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:12.675493 23934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:12.683889 23939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:12.683960 23942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:12.683876 23940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:12.683874 23934 server_base.cc:1034] running on GCE node
I20260927 15:19:12.684743 23934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:12.685928 23934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:12.687111 23934 hybrid_clock.cc:648] HybridClock initialized: now 1790522352687094 us; error 28 us; skew 500 ppm
Sep 27 15:19:12 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522352, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:12.697993 23934 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:12.698994 23934 webserver.cc:460] Webserver started at http://127.9.242.190:44695/ using document root <none> and password file <none>
I20260927 15:19:12.699576 23934 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:12.699651 23934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:12.699843 23934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:12.701983 23934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "ebe9ef9363d5425c8abc8e9a51743ac7"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "26e29f59816c46cd3eee67532ddcc215"
server_key_iv: "4924dd498bf452775d2fd6c5784ff1b5"
server_key_version: "encryptionkey@0"
I20260927 15:19:12.702528 23934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "ebe9ef9363d5425c8abc8e9a51743ac7"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "26e29f59816c46cd3eee67532ddcc215"
server_key_iv: "4924dd498bf452775d2fd6c5784ff1b5"
server_key_version: "encryptionkey@0"
I20260927 15:19:12.706182 23934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:12.708688 23949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:12.709972 23934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:12.710080 23934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "ebe9ef9363d5425c8abc8e9a51743ac7"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "26e29f59816c46cd3eee67532ddcc215"
server_key_iv: "4924dd498bf452775d2fd6c5784ff1b5"
server_key_version: "encryptionkey@0"
I20260927 15:19:12.710170 23934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:12.728374 23934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:12.731515 23934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:12.731691 23934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:12.738381 23934 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:39655
I20260927 15:19:12.738641 24001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:39655 every 8 connection(s)
I20260927 15:19:12.739365 23934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:12.742182 24002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:12.742461 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 23934
I20260927 15:19:12.742563 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:19:12.742822 10186 external_mini_cluster.cc:1428] Setting key 0cc8b573ab466ce714c44d7907f6e83f
I20260927 15:19:12.749087 24002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Bootstrap starting.
I20260927 15:19:12.751752 24002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Neither blocks nor log segments found. Creating new log.
Sep 27 15:19:12 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:12.752494 24002 log.cc:824] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:12.754640 24002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: No bootstrap required, opened a new log
I20260927 15:19:12.756984 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:12.744158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43942 (local address 127.9.242.190:39655)
0927 15:19:12.744575 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:12.744589 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:12.744830 (+ 241us) server_negotiation.cc:407] Connection header received
0927 15:19:12.745557 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:12.745575 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:12.746004 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:12.746294 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:12.747160 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:12.748126 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:12.749282 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:12.749776 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:12.749890 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:12.752454 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:12.752460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:12.752476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:12.752508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:12.754209 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:12.754699 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:12.754704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:12.754709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:12.754835 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:12.755113 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:12.755115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:12.755117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:12.755486 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:12.755684 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:12.755928 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:19:12.756205 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":116,"threads_started":1}
I20260927 15:19:12.758464 24002 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:12.758679 24002 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:12.758734 24002 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe9ef9363d5425c8abc8e9a51743ac7, State: Initialized, Role: FOLLOWER
I20260927 15:19:12.759183 24002 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:12.759341 24002 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:12.759403 24002 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:12.759491 24002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:12.760406 24002 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:12.760730 24002 leader_election.cc:302] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe9ef9363d5425c8abc8e9a51743ac7; no voters:
I20260927 15:19:12.761013 24002 leader_election.cc:288] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:12.761144 24007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:12.761358 24007 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 LEADER]: Becoming Leader. State: Replica: ebe9ef9363d5425c8abc8e9a51743ac7, State: Running, Role: LEADER
I20260927 15:19:12.761718 24007 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:12.762151 24002 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:12.763365 24009 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe9ef9363d5425c8abc8e9a51743ac7. Latest consensus state: current_term: 1 leader_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } } }
I20260927 15:19:12.763412 24008 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } } }
I20260927 15:19:12.763487 24009 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:12.763509 24008 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:12.763866 24018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:12.766584 24018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:12.771823 24018 catalog_manager.cc:1347] Generated new cluster ID: 18af19be2ce542b9a563587e558f7374
I20260927 15:19:12.771889 24018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:12.794938 24018 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:12.796314 24018 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:12.802898 24018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Generated new TSK 0
I20260927 15:19:12.803753 24018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:19:12.837080 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39655
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:12.948196 24028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:12.948462 24028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:12.948503 24028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:12.952605 24028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:12.952662 24028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:12.952728 24028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:19:12.958004 24028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:12.959028 24028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:12.960009 24028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:12.969385 24034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:12.969399 24036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:12.969643 24028 server_base.cc:1034] running on GCE node
W20260927 15:19:12.969432 24033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:12.970077 24028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:12.970709 24028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:12.971890 24028 hybrid_clock.cc:648] HybridClock initialized: now 1790522352971873 us; error 30 us; skew 500 ppm
Sep 27 15:19:12 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522352, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:12.974568 24028 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:19:12.975411 24028 webserver.cc:460] Webserver started at http://127.9.242.129:44557/ using document root <none> and password file <none>
I20260927 15:19:12.975910 24028 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:12.975961 24028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:12.976109 24028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:12.978039 24028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "73835b012405447faba430ac2b055bb0"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "5d70b5f9034203b2d936c05b3c11dbfb"
server_key_iv: "97348ebd7907c271fac5fb9bf9354d85"
server_key_version: "encryptionkey@0"
I20260927 15:19:12.978519 24028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "73835b012405447faba430ac2b055bb0"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "5d70b5f9034203b2d936c05b3c11dbfb"
server_key_iv: "97348ebd7907c271fac5fb9bf9354d85"
server_key_version: "encryptionkey@0"
I20260927 15:19:12.982121 24028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:19:12.984659 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:12.986137 24028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:12.986249 24028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "73835b012405447faba430ac2b055bb0"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "5d70b5f9034203b2d936c05b3c11dbfb"
server_key_iv: "97348ebd7907c271fac5fb9bf9354d85"
server_key_version: "encryptionkey@0"
I20260927 15:19:12.986363 24028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:12.999209 24028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:13.002470 24028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:13.002650 24028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:13.003283 24028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:13.004148 24028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:13.004194 24028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.004236 24028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:13.004266 24028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.012931 24028 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:36813
I20260927 15:19:13.012985 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:36813 every 8 connection(s)
I20260927 15:19:13.013907 24028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
Sep 27 15:19:13 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522352, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:13.023306 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24028
I20260927 15:19:13.023409 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:13.023666 10186 external_mini_cluster.cc:1428] Setting key 775a9fd329682998f31cea71163bf1d1
I20260927 15:19:13.028188 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.015722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:55707 (local address 127.9.242.190:39655)
0927 15:19:13.015856 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.015860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.016754 (+ 894us) server_negotiation.cc:407] Connection header received
0927 15:19:13.018150 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.018153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.018213 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.018291 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:13.019494 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.020024 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.021142 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.021374 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.021461 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.023967 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:13.023971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:13.023974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:13.024004 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:13.025765 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.026413 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.026417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.026419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.026506 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.026945 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.026949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.026951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.027218 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:13.027323 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.027860 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.027984 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:19:13.028942 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.016155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:55707)
0927 15:19:13.016600 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.016645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.017942 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.018473 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.018479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.018843 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:13.019340 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.019356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.020140 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.020143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.021026 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.021035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.021513 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.021517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.021716 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.022268 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:13.022284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:13.023785 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:13.025874 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:13.025881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:13.025900 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:13.026290 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:13.026593 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:13.026596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:13.026598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:13.026798 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:13.027348 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:13.027353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:13.027606 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.027792 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.028071 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260927 15:19:13.030431 24157 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39655
I20260927 15:19:13.030673 24157 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:13.031172 24157 heartbeater.cc:511] Master 127.9.242.190:39655 requested a full tablet report, sending...
I20260927 15:19:13.032914 23966 ts_manager.cc:194] Registered new tserver with Master: 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813)
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:19:13.034289 23966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:55707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:19:13.050508 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39655
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:13.159387 24162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:13.159647 24162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:13.159688 24162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:13.163794 24162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:13.163854 24162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:13.163913 24162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:19:13.169149 24162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:13.170192 24162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:13.171149 24162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:13.180253 24167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:13.180456 24162 server_base.cc:1034] running on GCE node
W20260927 15:19:13.180259 24170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:13.180655 24168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:13.181111 24162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:13.181926 24162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:13.183097 24162 hybrid_clock.cc:648] HybridClock initialized: now 1790522353183069 us; error 40 us; skew 500 ppm
Sep 27 15:19:13 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:13.185806 24162 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:19:13.186607 24162 webserver.cc:460] Webserver started at http://127.9.242.130:41369/ using document root <none> and password file <none>
I20260927 15:19:13.187139 24162 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:13.187196 24162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:13.187374 24162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:13.189368 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "2be118265b764a91a68ab3a3d23abb88"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "65fd1bfee6f98207948e30c992d3f0cb"
server_key_iv: "afa8b64bd2d3508e7b7791b9e1383860"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.189924 24162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "2be118265b764a91a68ab3a3d23abb88"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "65fd1bfee6f98207948e30c992d3f0cb"
server_key_iv: "afa8b64bd2d3508e7b7791b9e1383860"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.193511 24162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 15:19:13.195957 24177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.197361 24162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:13.197475 24162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "2be118265b764a91a68ab3a3d23abb88"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "65fd1bfee6f98207948e30c992d3f0cb"
server_key_iv: "afa8b64bd2d3508e7b7791b9e1383860"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.197592 24162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:13.213414 24162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:13.216413 24162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:13.216580 24162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:13.217132 24162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:13.217957 24162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:13.218001 24162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.218044 24162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:13.218072 24162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.226231 24162 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:40485
I20260927 15:19:13.226312 24290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:40485 every 8 connection(s)
I20260927 15:19:13.227119 24162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
Sep 27 15:19:13 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:13.236788 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24162
I20260927 15:19:13.236887 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:13.237125 10186 external_mini_cluster.cc:1428] Setting key 4fd731d4ccd3a82dbea41ae3b8f9dae1
I20260927 15:19:13.240561 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.229012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:35307 (local address 127.9.242.190:39655)
0927 15:19:13.229156 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.229160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.230041 (+ 881us) server_negotiation.cc:407] Connection header received
0927 15:19:13.231410 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.231413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.231471 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.231543 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:13.232746 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.233270 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.234441 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.234674 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.234770 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.237249 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:13.237253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:13.237256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:13.237285 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:13.238697 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.239159 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.239163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.239164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.239237 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.239538 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.239541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.239543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.239723 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:13.239803 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.240286 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.240405 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:19:13.241475 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.229407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:35307)
0927 15:19:13.229889 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.229934 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.231193 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.231734 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.231741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.232105 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:13.232602 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.232615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.233385 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.233388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.234326 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.234336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.234815 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.234819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.234982 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.235538 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:13.235555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:13.237080 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:13.238806 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:13.238813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:13.238827 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:13.239070 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:13.239316 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:13.239320 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:13.239322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:13.239455 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:13.239819 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:13.239825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:13.240060 (+ 235us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.240232 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.240557 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260927 15:19:13.242620 24291 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39655
I20260927 15:19:13.242893 24291 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:13.243474 24291 heartbeater.cc:511] Master 127.9.242.190:39655 requested a full tablet report, sending...
I20260927 15:19:13.244647 23966 ts_manager.cc:194] Registered new tserver with Master: 2be118265b764a91a68ab3a3d23abb88 (127.9.242.130:40485)
I20260927 15:19:13.245400 23966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:35307
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:19:13.262517 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39655
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:13.370000 24296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:13.370283 24296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:13.370327 24296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:13.374357 24296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:13.374418 24296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:13.374483 24296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:19:13.380033 24296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:13.381090 24296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:13.382169 24296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:13.391770 24304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:13.391772 24302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:13.391911 24296 server_base.cc:1034] running on GCE node
W20260927 15:19:13.391789 24301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:13.392553 24296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:13.393262 24296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:13.394436 24296 hybrid_clock.cc:648] HybridClock initialized: now 1790522353394404 us; error 44 us; skew 500 ppm
Sep 27 15:19:13 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:13.397388 24296 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:19:13.398406 24296 webserver.cc:460] Webserver started at http://127.9.242.131:33285/ using document root <none> and password file <none>
I20260927 15:19:13.399063 24296 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:13.399142 24296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:13.399350 24296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:13.401479 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "c20c9518c859415f8d6da1af1f90f600"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "dc5aeb53ff19ca21305604c927dbb357"
server_key_iv: "b62c47387254599a79e9bcae54fd0129"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.402081 24296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "c20c9518c859415f8d6da1af1f90f600"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "dc5aeb53ff19ca21305604c927dbb357"
server_key_iv: "b62c47387254599a79e9bcae54fd0129"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.405929 24296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:13.408509 24311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.409894 24296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:13.410017 24296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "c20c9518c859415f8d6da1af1f90f600"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "dc5aeb53ff19ca21305604c927dbb357"
server_key_iv: "b62c47387254599a79e9bcae54fd0129"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.410121 24296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:13.430135 24296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:13.433343 24296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:13.433528 24296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:13.434242 24296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:13.435072 24296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:13.435120 24296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.435174 24296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:13.435200 24296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.443485 24296 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:43709
I20260927 15:19:13.443532 24424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:43709 every 8 connection(s)
I20260927 15:19:13.444401 24296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:13.448657 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24296
I20260927 15:19:13.448770 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:13.449039 10186 external_mini_cluster.cc:1428] Setting key f670c179d533e00b1a7c2ee30df1997d
Sep 27 15:19:13 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:13.458178 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.446173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:59259 (local address 127.9.242.190:39655)
0927 15:19:13.446323 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.446328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.447319 (+ 991us) server_negotiation.cc:407] Connection header received
0927 15:19:13.448686 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.448689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.448748 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.448827 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:13.450424 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.451191 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.452351 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.452562 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.452651 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.455050 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:13.455053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:13.455062 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:13.455095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:13.456293 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.456784 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.456786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.456788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.456859 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.457147 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.457149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.457150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.457321 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:13.457390 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.457945 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.458045 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:19:13.458927 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.446615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:59259)
0927 15:19:13.447094 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.447197 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.448467 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.449077 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.449088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.449546 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:13.450292 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.450315 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.451302 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.451305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.452235 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.452245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.452684 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.452686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.452839 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.453373 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:13.453388 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:13.454893 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:13.456415 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:13.456420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:13.456432 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:13.456699 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:13.456929 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:13.456931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:13.456932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:13.457061 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:13.457400 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:13.457405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:13.457683 (+ 278us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.457862 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.458152 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260927 15:19:13.460031 24425 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39655
I20260927 15:19:13.460291 24425 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:13.460749 24425 heartbeater.cc:511] Master 127.9.242.190:39655 requested a full tablet report, sending...
I20260927 15:19:13.461872 23966 ts_manager.cc:194] Registered new tserver with Master: c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709)
I20260927 15:19:13.462497 23966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:59259
I20260927 15:19:13.462975 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:13.471357 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.464933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.9.242.190:39655)
0927 15:19:13.465062 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.465066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.465214 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:19:13.465296 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.465298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.465333 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.465391 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:13.466035 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.466544 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.467361 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.467570 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.467653 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.468812 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:13.468815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:13.468817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:13.468838 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:13.469993 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.470363 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.470366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.470367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.470433 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:13.470605 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:13.470607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:13.470608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:13.470814 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:13.470909 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.471086 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.471207 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:19:13.474076 23966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:13.476992 23966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:13.490947 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.485022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.0.0.1:57714)
0927 15:19:13.485934 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.485968 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.486098 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.486681 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.486686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.486708 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:13.486838 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.486845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.488403 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.488407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.489554 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.489563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.490476 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.490480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.490555 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.490572 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.490638 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.490784 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":33,"spinlock_wait_cycles":12544,"thread_start_us":99,"threads_started":1}
I20260927 15:19:13.491034 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.485173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.9.242.130:40485)
0927 15:19:13.485726 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.485738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.486227 (+ 489us) server_negotiation.cc:407] Connection header received
0927 15:19:13.486305 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.486309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.486503 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.486643 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:13.487035 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.487883 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.489396 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.490260 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.490361 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.490505 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.490788 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.490884 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":143,"threads_started":1}
I20260927 15:19:13.491312 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.486186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43709 (local address 127.0.0.1:36316)
0927 15:19:13.486427 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.486438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.486769 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.487259 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.487262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.487281 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:13.487354 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.487359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.488619 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.488623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.489712 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.489721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.490951 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.490955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.491034 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.491056 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.491120 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.491182 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":43,"thread_start_us":81,"threads_started":1}
I20260927 15:19:13.491307 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.485316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.9.242.129:36813)
0927 15:19:13.485620 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.485635 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.486057 (+ 422us) server_negotiation.cc:407] Connection header received
0927 15:19:13.486286 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.486293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.486506 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.486973 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:13.487132 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.488273 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.489689 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.490353 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.490871 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.491009 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.491059 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.491168 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":52,"threads_started":1}
I20260927 15:19:13.491737 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.485935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.9.242.131:43709)
0927 15:19:13.486626 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.486637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.486657 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:19:13.486917 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.486922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.487123 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.487267 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:13.487462 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.488499 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.489875 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.490627 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.490721 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.490852 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.491511 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.491608 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":92,"threads_started":1}
I20260927 15:19:13.491523 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.484989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40485 (local address 127.0.0.1:53232)
0927 15:19:13.485935 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.485970 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.486098 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.486817 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.486820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.486837 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:13.486934 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.486939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.487996 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.488000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.489271 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.489281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.490377 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.490380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.490511 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.490555 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.490667 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.490784 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":66,"spinlock_wait_cycles":5248,"thread_start_us":64,"threads_started":1}
I20260927 15:19:13.494560 24091 tablet_service.cc:1467] Processing CreateTablet for tablet c58f8ccf3f844fcf9123058ac47a99d8 (DEFAULT_TABLE table=test-table [id=fcd90fcc4c0c47939a1598202cdcc327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:13.494560 24359 tablet_service.cc:1467] Processing CreateTablet for tablet c58f8ccf3f844fcf9123058ac47a99d8 (DEFAULT_TABLE table=test-table [id=fcd90fcc4c0c47939a1598202cdcc327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:13.495806 24359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58f8ccf3f844fcf9123058ac47a99d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:13.495800 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58f8ccf3f844fcf9123058ac47a99d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:13.499451 24225 tablet_service.cc:1467] Processing CreateTablet for tablet c58f8ccf3f844fcf9123058ac47a99d8 (DEFAULT_TABLE table=test-table [id=fcd90fcc4c0c47939a1598202cdcc327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:13.500412 24225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58f8ccf3f844fcf9123058ac47a99d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:13.503400 24443 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Bootstrap starting.
I20260927 15:19:13.503947 24442 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Bootstrap starting.
I20260927 15:19:13.506362 24442 tablet_bootstrap.cc:654] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:13.507701 24442 log.cc:824] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:13.507802 24444 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Bootstrap starting.
I20260927 15:19:13.509858 24443 tablet_bootstrap.cc:654] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:13.509869 24442 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: No bootstrap required, opened a new log
I20260927 15:19:13.510047 24442 ts_tablet_manager.cc:1397] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:19:13.510556 24444 tablet_bootstrap.cc:654] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:13.511075 24443 log.cc:824] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:13.511415 24444 log.cc:824] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:13.512795 24443 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: No bootstrap required, opened a new log
I20260927 15:19:13.512984 24443 ts_tablet_manager.cc:1397] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 15:19:13.513284 24444 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: No bootstrap required, opened a new log
I20260927 15:19:13.513454 24444 ts_tablet_manager.cc:1397] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:19:13.513875 24442 raft_consensus.cc:348] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.514299 24442 raft_consensus.cc:374] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:13.514372 24442 raft_consensus.cc:729] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20c9518c859415f8d6da1af1f90f600, State: Initialized, Role: FOLLOWER
I20260927 15:19:13.514914 24442 consensus_queue.cc:260] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.515578 24425 heartbeater.cc:503] Master 127.9.242.190:39655 was elected leader, sending a full tablet report...
I20260927 15:19:13.515910 24442 ts_tablet_manager.cc:1428] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:19:13.516924 24444 raft_consensus.cc:348] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.517120 24444 raft_consensus.cc:374] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:13.517195 24444 raft_consensus.cc:729] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be118265b764a91a68ab3a3d23abb88, State: Initialized, Role: FOLLOWER
I20260927 15:19:13.517377 24443 raft_consensus.cc:348] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.517629 24443 raft_consensus.cc:374] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:13.517580 24444 consensus_queue.cc:260] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.517717 24443 raft_consensus.cc:729] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73835b012405447faba430ac2b055bb0, State: Initialized, Role: FOLLOWER
I20260927 15:19:13.518185 24291 heartbeater.cc:503] Master 127.9.242.190:39655 was elected leader, sending a full tablet report...
I20260927 15:19:13.518157 24443 consensus_queue.cc:260] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.518496 24444 ts_tablet_manager.cc:1428] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:19:13.518999 24157 heartbeater.cc:503] Master 127.9.242.190:39655 was elected leader, sending a full tablet report...
I20260927 15:19:13.519068 24443 ts_tablet_manager.cc:1428] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:19:13.616016 24449 raft_consensus.cc:482] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:13.616236 24449 raft_consensus.cc:504] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.617377 24449 leader_election.cc:288] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813), c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709)
I20260927 15:19:13.621235 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.617759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:40087 (local address 127.9.242.129:36813)
0927 15:19:13.617883 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.617887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.617903 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:19:13.618002 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.618006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.618060 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.618144 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:13.618436 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.619076 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.620268 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.620695 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.620792 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.620831 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.621032 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.621102 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:19:13.621241 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.617604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.9.242.130:40087)
0927 15:19:13.617805 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.617819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.617903 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.618201 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.618204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.618224 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:13.618322 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.618327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.619179 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.619183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.620146 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.620155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.620805 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.620809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.620915 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.620939 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.621007 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.621075 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":51}
I20260927 15:19:13.621320 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.618030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:41213 (local address 127.9.242.131:43709)
0927 15:19:13.618170 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.618175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.618383 (+ 208us) server_negotiation.cc:407] Connection header received
0927 15:19:13.618485 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.618487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.618525 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.618579 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:13.618822 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.619376 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.620416 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.620839 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.620940 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.620984 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.621152 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.621217 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:19:13.621534 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.617898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43709 (local address 127.9.242.130:41213)
0927 15:19:13.618302 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.618313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:13.618397 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:13.618599 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:13.618602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:13.618619 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:13.618728 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.618733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.619469 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.619472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.620303 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:13.620312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.620940 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.620942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:13.621036 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.621058 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:19:13.621128 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:19:13.621192 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":22,"thread_start_us":138,"threads_started":1}
I20260927 15:19:13.621920 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58f8ccf3f844fcf9123058ac47a99d8" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73835b012405447faba430ac2b055bb0" is_pre_election: true
I20260927 15:19:13.622208 24111 raft_consensus.cc:2454] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 0.
I20260927 15:19:13.622565 24379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58f8ccf3f844fcf9123058ac47a99d8" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20c9518c859415f8d6da1af1f90f600" is_pre_election: true
I20260927 15:19:13.622665 24178 leader_election.cc:302] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be118265b764a91a68ab3a3d23abb88, 73835b012405447faba430ac2b055bb0; no voters:
I20260927 15:19:13.622830 24379 raft_consensus.cc:2454] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 0.
I20260927 15:19:13.622944 24449 raft_consensus.cc:2781] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:13.623034 24449 raft_consensus.cc:482] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:13.623083 24449 raft_consensus.cc:3037] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:13.624107 24449 raft_consensus.cc:504] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.624432 24449 leader_election.cc:288] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 election: Requested vote from peers 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813), c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709)
I20260927 15:19:13.624847 24379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58f8ccf3f844fcf9123058ac47a99d8" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20c9518c859415f8d6da1af1f90f600"
I20260927 15:19:13.624898 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58f8ccf3f844fcf9123058ac47a99d8" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73835b012405447faba430ac2b055bb0"
I20260927 15:19:13.624946 24379 raft_consensus.cc:3037] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:13.625029 24111 raft_consensus.cc:3037] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:13.626019 24379 raft_consensus.cc:2454] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 1.
I20260927 15:19:13.626101 24111 raft_consensus.cc:2454] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 1.
I20260927 15:19:13.626260 24178 leader_election.cc:302] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be118265b764a91a68ab3a3d23abb88, c20c9518c859415f8d6da1af1f90f600; no voters:
I20260927 15:19:13.626415 24449 raft_consensus.cc:2781] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:13.626616 24449 raft_consensus.cc:686] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 1 LEADER]: Becoming Leader. State: Replica: 2be118265b764a91a68ab3a3d23abb88, State: Running, Role: LEADER
I20260927 15:19:13.626916 24449 consensus_queue.cc:237] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:13.629989 23966 catalog_manager.cc:5495] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2be118265b764a91a68ab3a3d23abb88 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "2be118265b764a91a68ab3a3d23abb88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:13.648406 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.645064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.9.242.130:40485)
0927 15:19:13.645190 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:13.645193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:13.645206 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:19:13.645281 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:13.645283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:13.645334 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:13.645419 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:13.645609 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.646165 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.647023 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:13.647237 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:13.647315 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:13.647594 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:13.648008 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:13.648091 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:13.648221 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 15:19:13.648270 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:19:13.654443 24111 raft_consensus.cc:1261] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Refusing update from remote peer 2be118265b764a91a68ab3a3d23abb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:13.654443 24379 raft_consensus.cc:1261] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Refusing update from remote peer 2be118265b764a91a68ab3a3d23abb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:13.655230 24449 consensus_queue.cc:1035] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:13.655427 24452 consensus_queue.cc:1035] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:13.663828 24457 mvcc.cc:205] Tried to move back new op lower bound from 7333979560560668672 to 7333979560458301440. Current Snapshot: MvccSnapshot[applied={T|T < 7333979560560668672}]
I20260927 15:19:13.673358 23966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:13.673521 23966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:13.675746 23966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Sending DeleteTablet for 3 replicas of tablet c58f8ccf3f844fcf9123058ac47a99d8
I20260927 15:19:13.676422 24225 tablet_service.cc:1514] Processing DeleteTablet for tablet c58f8ccf3f844fcf9123058ac47a99d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:13 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:53232
I20260927 15:19:13.676472 24091 tablet_service.cc:1514] Processing DeleteTablet for tablet c58f8ccf3f844fcf9123058ac47a99d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:13 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:57714
I20260927 15:19:13.676649 24359 tablet_service.cc:1514] Processing DeleteTablet for tablet c58f8ccf3f844fcf9123058ac47a99d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:13 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:36316
I20260927 15:19:13.677028 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 24028
I20260927 15:19:13.677037 24466 tablet_replica.cc:330] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: stopping tablet replica
I20260927 15:19:13.677145 24465 tablet_replica.cc:330] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: stopping tablet replica
I20260927 15:19:13.677309 24466 raft_consensus.cc:2229] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:13.677404 24465 raft_consensus.cc:2229] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:13.677798 24466 raft_consensus.cc:2258] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:13.678906 24467 tablet_replica.cc:330] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: stopping tablet replica
I20260927 15:19:13.679199 24467 raft_consensus.cc:2229] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:13.679529 24467 raft_consensus.cc:2258] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:13.680413 24466 ts_tablet_manager.cc:1905] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:13.680838 24467 ts_tablet_manager.cc:1905] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:13.683157 24466 ts_tablet_manager.cc:1918] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:13.683308 24466 log.cc:1196] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/c58f8ccf3f844fcf9123058ac47a99d8
I20260927 15:19:13.683440 24467 ts_tablet_manager.cc:1918] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:13.683585 24467 log.cc:1196] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/c58f8ccf3f844fcf9123058ac47a99d8
W20260927 15:19:13.683720 23950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:36813 (error 108)
I20260927 15:19:13.683795 24466 ts_tablet_manager.cc:1939] T c58f8ccf3f844fcf9123058ac47a99d8 P 2be118265b764a91a68ab3a3d23abb88: Deleting consensus metadata
I20260927 15:19:13.683912 24467 ts_tablet_manager.cc:1939] T c58f8ccf3f844fcf9123058ac47a99d8 P c20c9518c859415f8d6da1af1f90f600: Deleting consensus metadata
I20260927 15:19:13.684568 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 24162
I20260927 15:19:13.686147 23950 catalog_manager.cc:4841] TS c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709): tablet c58f8ccf3f844fcf9123058ac47a99d8 (table test-table [id=fcd90fcc4c0c47939a1598202cdcc327]) successfully deleted
I20260927 15:19:13.686264 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.686006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.0.0.1:57726)
0927 15:19:13.686112 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.686177 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 15:19:13.686371 23950 catalog_manager.cc:4584] TS 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c58f8ccf3f844fcf9123058ac47a99d8: Network error: Client connection negotiation failed: client connection to 127.9.242.129:36813: connect: Connection refused (error 111)
I20260927 15:19:13.689462 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 24296
I20260927 15:19:13.691335 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:13.691085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40485 (local address 127.0.0.1:53248)
0927 15:19:13.691210 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:13.691261 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.130:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 15:19:13.694298 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 23934
I20260927 15:19:13.699270 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39655
--webserver_interface=127.9.242.190
--webserver_port=44695
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:39655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:13.807319 24469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:13.807607 24469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:13.807664 24469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:13.811875 24469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:13.811936 24469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:13.811964 24469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:13.811985 24469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:13.812019 24469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:13.817301 24469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:39655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:13.818349 24469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:13.819319 24469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:13.831890 24476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:13.831933 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:13.831912 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:13.832252 24469 server_base.cc:1034] running on GCE node
I20260927 15:19:13.832875 24469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:13.834218 24469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:13.835390 24469 hybrid_clock.cc:648] HybridClock initialized: now 1790522353835375 us; error 38 us; skew 500 ppm
Sep 27 15:19:13 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522353, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:13.838290 24469 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:13.839216 24469 webserver.cc:460] Webserver started at http://127.9.242.190:44695/ using document root <none> and password file <none>
I20260927 15:19:13.839829 24469 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:13.839902 24469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:13.843851 24469 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:19:13.846058 24485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:13.847321 24469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:13.847430 24469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "ebe9ef9363d5425c8abc8e9a51743ac7"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "26e29f59816c46cd3eee67532ddcc215"
server_key_iv: "4924dd498bf452775d2fd6c5784ff1b5"
server_key_version: "encryptionkey@0"
I20260927 15:19:13.848021 24469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:13.869689 24469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:13.872568 24469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:13.872725 24469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:13.879264 24469 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:39655
I20260927 15:19:13.879513 24537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:39655 every 8 connection(s)
I20260927 15:19:13.880326 24469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:13.884874 24538 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:19:13.885309 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24469
I20260927 15:19:13.885790 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:36813
--local_ip_for_outbound_sockets=127.9.242.129
--tserver_master_addrs=127.9.242.190:39655
--webserver_port=44557
--webserver_interface=127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:13.886945 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Bootstrap starting.
I20260927 15:19:13.903820 24538 log.cc:824] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:13.912467 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:13.912823 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Bootstrap complete.
I20260927 15:19:13.916664 24538 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:13.917052 24538 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe9ef9363d5425c8abc8e9a51743ac7, State: Initialized, Role: FOLLOWER
I20260927 15:19:13.917529 24538 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:13.917690 24538 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:13.917795 24538 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:13.917899 24538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:19:13.919777 24538 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:13.920078 24538 leader_election.cc:302] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe9ef9363d5425c8abc8e9a51743ac7; no voters:
I20260927 15:19:13.920363 24538 leader_election.cc:288] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:19:13.920480 24542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:19:13.920677 24542 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 LEADER]: Becoming Leader. State: Replica: ebe9ef9363d5425c8abc8e9a51743ac7, State: Running, Role: LEADER
I20260927 15:19:13.921017 24542 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:13.921554 24538 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:13.922335 24543 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } } }
I20260927 15:19:13.922453 24543 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:13.922732 24544 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe9ef9363d5425c8abc8e9a51743ac7. Latest consensus state: current_term: 2 leader_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } } }
I20260927 15:19:13.922847 24544 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:13.923162 24550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:13.927762 24550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:13.928486 24550 catalog_manager.cc:1259] Loaded cluster ID: 18af19be2ce542b9a563587e558f7374
I20260927 15:19:13.928546 24550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:13.930033 24550 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 15:19:13.934782 24550 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 15:19:13.940039 24550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:13.940776 24550 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Loaded TSK: 0
I20260927 15:19:13.941989 24550 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:19:13.996912 24540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:13.997200 24540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:13.997263 24540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:14.001428 24540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:14.001497 24540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:14.001582 24540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:19:14.007331 24540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:36813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=44557
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:14.008396 24540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:14.009446 24540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:14.019126 24565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.019140 24568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.019230 24566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:14.019567 24540 server_base.cc:1034] running on GCE node
I20260927 15:19:14.020123 24540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:14.020973 24540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:14.022166 24540 hybrid_clock.cc:648] HybridClock initialized: now 1790522354022132 us; error 48 us; skew 500 ppm
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:14.025542 24540 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:19:14.026681 24540 webserver.cc:460] Webserver started at http://127.9.242.129:44557/ using document root <none> and password file <none>
I20260927 15:19:14.027405 24540 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:14.027503 24540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:14.032645 24540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:19:14.035367 24575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.036891 24540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:19:14.037025 24540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "73835b012405447faba430ac2b055bb0"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "5d70b5f9034203b2d936c05b3c11dbfb"
server_key_iv: "97348ebd7907c271fac5fb9bf9354d85"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.037781 24540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:14.061691 24540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:14.065383 24540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:14.065613 24540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:14.066382 24540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:14.067687 24582 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:19:14.071668 24540 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:19:14.071736 24540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 15:19:14.071797 24540 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:19:14.073890 24540 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:19:14.073945 24540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:14.074258 24582 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Bootstrap starting.
I20260927 15:19:14.083042 24540 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:36813
I20260927 15:19:14.083132 24689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:36813 every 8 connection(s)
I20260927 15:19:14.083968 24540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:14.092448 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24540
I20260927 15:19:14.092926 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:40485
--local_ip_for_outbound_sockets=127.9.242.130
--tserver_master_addrs=127.9.242.190:39655
--webserver_port=41369
--webserver_interface=127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:14.094120 24582 log.cc:824] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:14.100936 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.086030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:60215)
0927 15:19:14.086496 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.086546 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.087375 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.088807 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.088815 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.089186 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:14.089711 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.089732 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.091778 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.091781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.092811 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.092822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.093679 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.093684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.093864 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.094416 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:14.094432 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:14.095978 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:14.098059 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.098064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.098078 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.098316 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.098649 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.098652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.098654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.098784 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.099321 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:14.099327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:14.099556 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.099721 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.100042 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260927 15:19:14.100924 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.086063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:60215 (local address 127.9.242.190:39655)
0927 15:19:14.086590 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.086608 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.086655 (+ 47us) server_negotiation.cc:407] Connection header received
0927 15:19:14.087831 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.087847 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.088282 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.088702 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:14.089902 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.091598 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.092931 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.093507 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.093684 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.096182 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:14.096187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:14.096211 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:14.096243 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:14.097951 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.098422 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.098428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.098434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.098554 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.098868 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.098872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.098873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.099214 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:14.099329 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.099845 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.100098 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":131,"threads_started":1}
I20260927 15:19:14.103164 24690 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39655
I20260927 15:19:14.103387 24690 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:14.103576 24582 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:14.103914 24690 heartbeater.cc:511] Master 127.9.242.190:39655 requested a full tablet report, sending...
I20260927 15:19:14.103967 24582 tablet_bootstrap.cc:492] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Bootstrap complete.
I20260927 15:19:14.105283 24582 ts_tablet_manager.cc:1397] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.008s
I20260927 15:19:14.106181 24502 ts_manager.cc:194] Registered new tserver with Master: 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813)
I20260927 15:19:14.108137 24582 raft_consensus.cc:348] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:14.108630 24582 raft_consensus.cc:729] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73835b012405447faba430ac2b055bb0, State: Initialized, Role: FOLLOWER
I20260927 15:19:14.109138 24582 consensus_queue.cc:260] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } } peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } }
I20260927 15:19:14.109807 24502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:60215
I20260927 15:19:14.110110 24582 ts_tablet_manager.cc:1428] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260927 15:19:14.112030 24690 heartbeater.cc:503] Master 127.9.242.190:39655 was elected leader, sending a full tablet report...
I20260927 15:19:14.116254 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.109495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.9.242.129:36813)
0927 15:19:14.109778 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.109788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.112327 (+ 2539us) server_negotiation.cc:407] Connection header received
0927 15:19:14.112379 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.112382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.112555 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.112698 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:14.112961 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.113818 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.115056 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.115609 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.115709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.115848 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.116025 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.116118 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260927 15:19:14.116297 24696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.108797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.0.0.1:57742)
0927 15:19:14.110663 (+ 1866us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.110695 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.110816 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.112709 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.112722 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.112747 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:14.112864 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.112870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.113928 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.113932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.114859 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.114869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.115727 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.115731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.115847 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.115886 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.115986 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.116171 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"thread_start_us":116,"threads_started":1}
I20260927 15:19:14.117326 24624 tablet_service.cc:1514] Processing DeleteTablet for tablet c58f8ccf3f844fcf9123058ac47a99d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:13 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:57742
I20260927 15:19:14.118278 24699 tablet_replica.cc:330] stopping tablet replica
I20260927 15:19:14.118544 24699 raft_consensus.cc:2229] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:14.118769 24699 raft_consensus.cc:2258] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:14.119633 24699 ts_tablet_manager.cc:1905] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:14.123315 24699 ts_tablet_manager.cc:1918] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:14.123404 24699 log.cc:1196] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/c58f8ccf3f844fcf9123058ac47a99d8
I20260927 15:19:14.123639 24699 ts_tablet_manager.cc:1939] T c58f8ccf3f844fcf9123058ac47a99d8 P 73835b012405447faba430ac2b055bb0: Deleting consensus metadata
I20260927 15:19:14.125468 24486 catalog_manager.cc:4841] TS 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813): tablet c58f8ccf3f844fcf9123058ac47a99d8 (table test-table [id=fcd90fcc4c0c47939a1598202cdcc327]) successfully deleted
W20260927 15:19:14.209933 24694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:14.210182 24694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:14.210224 24694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:14.214337 24694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:14.214406 24694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:14.214463 24694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:19:14.220139 24694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:40485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=41369
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:14.221259 24694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:14.222378 24694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:14.232092 24705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.232080 24707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.232081 24704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:14.232478 24694 server_base.cc:1034] running on GCE node
I20260927 15:19:14.232906 24694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:14.233597 24694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:14.234820 24694 hybrid_clock.cc:648] HybridClock initialized: now 1790522354234800 us; error 31 us; skew 500 ppm
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:14.237581 24694 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:19:14.238497 24694 webserver.cc:460] Webserver started at http://127.9.242.130:41369/ using document root <none> and password file <none>
I20260927 15:19:14.239002 24694 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:14.239068 24694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:14.242959 24694 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:14.245074 24714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.246354 24694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:14.246451 24694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "2be118265b764a91a68ab3a3d23abb88"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "65fd1bfee6f98207948e30c992d3f0cb"
server_key_iv: "afa8b64bd2d3508e7b7791b9e1383860"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.246966 24694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:14.258762 24694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:14.261452 24694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:14.261631 24694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:14.262190 24694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:14.262943 24694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:14.262985 24694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.263022 24694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:14.263043 24694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.271579 24694 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:40485
I20260927 15:19:14.271607 24827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:40485 every 8 connection(s)
I20260927 15:19:14.272531 24694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:14.280565 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24694
I20260927 15:19:14.280975 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:43709
--local_ip_for_outbound_sockets=127.9.242.131
--tserver_master_addrs=127.9.242.190:39655
--webserver_port=33285
--webserver_interface=127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:14.286365 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.274239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:54055 (local address 127.9.242.190:39655)
0927 15:19:14.274360 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.274363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.275194 (+ 831us) server_negotiation.cc:407] Connection header received
0927 15:19:14.276523 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.276526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.276626 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.276698 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:14.277949 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.278470 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.279603 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.279844 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.279934 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.282756 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:14.282759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:14.282761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:14.282787 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:14.284081 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.284651 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.284654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.284656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.284727 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.285176 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.285179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.285180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.285384 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:14.285472 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.286123 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.286241 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:19:14.287523 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.274569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:54055)
0927 15:19:14.275044 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.275084 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.276306 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.276877 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.276883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.277231 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:14.277803 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.277819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.278585 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.278588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.279481 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.279491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.279983 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.279987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.280149 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.280788 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:14.280809 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:14.282569 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:14.284194 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.284200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.284217 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.284560 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.284817 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.284820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.284823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.285006 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.285492 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:14.285499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:14.285853 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.286093 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.286455 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":158,"threads_started":1}
I20260927 15:19:14.288738 24828 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39655
I20260927 15:19:14.288962 24828 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:14.289426 24828 heartbeater.cc:511] Master 127.9.242.190:39655 requested a full tablet report, sending...
I20260927 15:19:14.290412 24502 ts_manager.cc:194] Registered new tserver with Master: 2be118265b764a91a68ab3a3d23abb88 (127.9.242.130:40485)
I20260927 15:19:14.290952 24502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:54055
W20260927 15:19:14.390647 24831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:14.390980 24831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:14.391047 24831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:14.395074 24831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:14.395138 24831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:14.395219 24831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:19:14.400556 24831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:43709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=33285
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:14.401628 24831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:14.402659 24831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:14.411993 24836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.411998 24839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.411990 24837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:14.412179 24831 server_base.cc:1034] running on GCE node
I20260927 15:19:14.412631 24831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:14.413291 24831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:14.414450 24831 hybrid_clock.cc:648] HybridClock initialized: now 1790522354414427 us; error 35 us; skew 500 ppm
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:14.417140 24831 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:19:14.418067 24831 webserver.cc:460] Webserver started at http://127.9.242.131:33285/ using document root <none> and password file <none>
I20260927 15:19:14.418588 24831 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:14.418663 24831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:14.422459 24831 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:14.424662 24846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.426044 24831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:14.426153 24831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "c20c9518c859415f8d6da1af1f90f600"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "dc5aeb53ff19ca21305604c927dbb357"
server_key_iv: "b62c47387254599a79e9bcae54fd0129"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.426714 24831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:14.439363 24831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:14.442327 24831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:14.442494 24831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:14.443125 24831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:14.443953 24831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:14.444000 24831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.444049 24831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:14.444077 24831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.452423 24831 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:43709
I20260927 15:19:14.452463 24959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:43709 every 8 connection(s)
I20260927 15:19:14.453372 24831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:14.457273 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24831
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:14.466207 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.457939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.9.242.190:39655)
0927 15:19:14.458183 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.458187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.458263 (+ 76us) server_negotiation.cc:407] Connection header received
0927 15:19:14.458369 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.458373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.458419 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.458500 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:14.459169 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.459743 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.460594 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.460918 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.460993 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.462450 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:14.462454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:14.462457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:14.462483 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:14.464522 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.464967 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.464970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.464972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.465048 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.465344 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.465347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.465348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.465566 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:14.465807 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.465918 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.466023 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260927 15:19:14.467501 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.455119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:45415 (local address 127.9.242.190:39655)
0927 15:19:14.455281 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.455285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.456130 (+ 845us) server_negotiation.cc:407] Connection header received
0927 15:19:14.457629 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.457632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.457723 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.457807 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:14.459188 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.459731 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.460918 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.461196 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.461294 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.463850 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:14.463854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:14.463857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:14.463892 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:14.465292 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.465931 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.465935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.465936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.466016 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:14.466408 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:14.466411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:14.466413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:14.466587 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:14.466678 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.467238 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.467360 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":72}
I20260927 15:19:14.468338 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.455500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:45415)
0927 15:19:14.455955 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.455997 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.457402 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.458014 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.458024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.458521 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:14.459069 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.459082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.459842 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.459846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.460805 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.460815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.461358 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.461363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.461562 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.462082 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:14.462098 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:14.463694 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:14.465395 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.465401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.465417 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.465838 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.466104 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:14.466107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:14.466110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:14.466292 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:14.466769 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:14.466775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:14.467012 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.467191 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.467490 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":123,"threads_started":1}
I20260927 15:19:14.469391 24960 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39655
I20260927 15:19:14.469624 24960 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:14.470111 24960 heartbeater.cc:511] Master 127.9.242.190:39655 requested a full tablet report, sending...
I20260927 15:19:14.471064 24502 ts_manager.cc:194] Registered new tserver with Master: c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709)
I20260927 15:19:14.471601 24502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:45415
I20260927 15:19:14.479583 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:14.483592 24693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.480129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.9.242.190:39655)
0927 15:19:14.480242 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.480245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.480393 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:19:14.480536 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.480538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.480634 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.480711 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:14.480889 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.481404 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.482284 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.482516 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.482614 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.482884 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:14.483291 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:14.483380 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.483430 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.483476 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 15:19:14.484179 24502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:14.487448 24502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:14.494778 24624 tablet_service.cc:1467] Processing CreateTablet for tablet 661a7931c91f43e1997e4d7c6c7b98a9 (DEFAULT_TABLE table=test-table [id=59015dede3c14e4a9dbce1514baf79fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:14.495409 24624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661a7931c91f43e1997e4d7c6c7b98a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:14.498431 24582 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Bootstrap starting.
I20260927 15:19:14.499094 24696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.493764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43709 (local address 127.0.0.1:36318)
0927 15:19:14.493920 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.493934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.494001 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.495117 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.495119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.495137 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:14.495244 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.495249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.496513 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.496516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.497714 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.497725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.498717 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.498720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.498809 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.498836 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.498880 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.498956 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":28}
I20260927 15:19:14.499267 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.494280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40485 (local address 127.0.0.1:53262)
0927 15:19:14.494591 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.494603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.494684 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.495420 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.495423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.495439 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:14.495522 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.495528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.496558 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.496562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.497607 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.497616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.498839 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.498843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.498936 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.498961 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.499053 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.499122 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":64,"thread_start_us":82,"threads_started":1}
I20260927 15:19:14.499333 24582 tablet_bootstrap.cc:654] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:14.499789 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.494012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.9.242.131:43709)
0927 15:19:14.494391 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.494404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.494428 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:19:14.494621 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.494628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.494849 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.495025 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:14.495357 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.496367 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.497832 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.498583 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.499364 (+ 781us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.499506 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.499558 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.499654 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":169,"threads_started":1}
I20260927 15:19:14.500159 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.494600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.9.242.130:40485)
0927 15:19:14.494991 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.495002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.495024 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:19:14.495095 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.495101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.495302 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.495445 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:14.495648 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.496413 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.497745 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.498710 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.498826 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.498980 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.499085 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.499760 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260927 15:19:14.500367 24582 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: No bootstrap required, opened a new log
I20260927 15:19:14.500454 24582 ts_tablet_manager.cc:1397] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:19:14.501057 24582 raft_consensus.cc:348] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.501173 24582 raft_consensus.cc:374] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:14.501209 24582 raft_consensus.cc:729] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73835b012405447faba430ac2b055bb0, State: Initialized, Role: FOLLOWER
I20260927 15:19:14.501331 24582 consensus_queue.cc:260] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.501591 24582 ts_tablet_manager.cc:1428] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 15:19:14.502825 24762 tablet_service.cc:1467] Processing CreateTablet for tablet 661a7931c91f43e1997e4d7c6c7b98a9 (DEFAULT_TABLE table=test-table [id=59015dede3c14e4a9dbce1514baf79fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:14.502826 24894 tablet_service.cc:1467] Processing CreateTablet for tablet 661a7931c91f43e1997e4d7c6c7b98a9 (DEFAULT_TABLE table=test-table [id=59015dede3c14e4a9dbce1514baf79fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:14.503861 24762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661a7931c91f43e1997e4d7c6c7b98a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:14.503872 24894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661a7931c91f43e1997e4d7c6c7b98a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:14.510659 24973 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Bootstrap starting.
I20260927 15:19:14.510712 24974 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Bootstrap starting.
I20260927 15:19:14.512964 24974 tablet_bootstrap.cc:654] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:14.512970 24973 tablet_bootstrap.cc:654] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:14.513638 24974 log.cc:824] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:14.513638 24973 log.cc:824] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:14.515460 24973 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: No bootstrap required, opened a new log
I20260927 15:19:14.515607 24974 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: No bootstrap required, opened a new log
I20260927 15:19:14.515627 24973 ts_tablet_manager.cc:1397] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 15:19:14.515769 24974 ts_tablet_manager.cc:1397] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 15:19:14.518419 24973 raft_consensus.cc:348] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.518419 24974 raft_consensus.cc:348] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.518616 24973 raft_consensus.cc:374] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:14.518618 24974 raft_consensus.cc:374] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:14.518647 24973 raft_consensus.cc:729] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be118265b764a91a68ab3a3d23abb88, State: Initialized, Role: FOLLOWER
I20260927 15:19:14.518661 24974 raft_consensus.cc:729] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20c9518c859415f8d6da1af1f90f600, State: Initialized, Role: FOLLOWER
I20260927 15:19:14.519048 24973 consensus_queue.cc:260] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.519048 24974 consensus_queue.cc:260] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.519865 24960 heartbeater.cc:503] Master 127.9.242.190:39655 was elected leader, sending a full tablet report...
I20260927 15:19:14.519974 24973 ts_tablet_manager.cc:1428] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260927 15:19:14.519979 24974 ts_tablet_manager.cc:1428] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260927 15:19:14.520087 24828 heartbeater.cc:503] Master 127.9.242.190:39655 was elected leader, sending a full tablet report...
W20260927 15:19:14.523351 24829 tablet.cc:2369] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:14.569574 24977 raft_consensus.cc:482] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:14.569872 24977 raft_consensus.cc:504] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.570973 24977 leader_election.cc:288] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709), 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813)
I20260927 15:19:14.574389 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.571342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:44063 (local address 127.9.242.131:43709)
0927 15:19:14.571472 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.571476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.571492 (+ 16us) server_negotiation.cc:407] Connection header received
0927 15:19:14.571562 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.571564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.571609 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.571672 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:14.571932 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.572466 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.573472 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.573889 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.573994 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.574049 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.574215 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.574285 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:19:14.574389 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.571236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43709 (local address 127.9.242.130:44063)
0927 15:19:14.571357 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.571371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.571474 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.571695 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.571697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.571713 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:14.571826 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.571831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.572574 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.572577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.573348 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.573356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.573994 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.573998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.574103 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.574124 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.574187 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.574256 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 15:19:14.575181 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.571634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:47685 (local address 127.9.242.129:36813)
0927 15:19:14.571755 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.571760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.571984 (+ 224us) server_negotiation.cc:407] Connection header received
0927 15:19:14.572077 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.572080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.572125 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.572185 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:14.572429 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.572987 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.574038 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.574416 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.574523 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.574577 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.574960 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.575046 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:19:14.575101 24914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661a7931c91f43e1997e4d7c6c7b98a9" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20c9518c859415f8d6da1af1f90f600" is_pre_election: true
I20260927 15:19:14.575433 24914 raft_consensus.cc:2454] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 0.
I20260927 15:19:14.575524 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.571532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.9.242.130:47685)
0927 15:19:14.571892 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.571902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:14.571993 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:14.572206 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:14.572208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:14.572223 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:14.572333 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.572338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.573101 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.573104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.573928 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:14.573937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.574633 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.574637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:14.574739 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.574763 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:14.575318 (+ 555us) client_negotiation.cc:240] Negotiation successful
0927 15:19:14.575391 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260927 15:19:14.575868 24715 leader_election.cc:302] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be118265b764a91a68ab3a3d23abb88, c20c9518c859415f8d6da1af1f90f600; no voters:
I20260927 15:19:14.576109 24977 raft_consensus.cc:2781] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:14.576092 24644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661a7931c91f43e1997e4d7c6c7b98a9" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73835b012405447faba430ac2b055bb0" is_pre_election: true
I20260927 15:19:14.576198 24977 raft_consensus.cc:482] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:14.576231 24977 raft_consensus.cc:3037] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:14.576354 24644 raft_consensus.cc:2454] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 0.
I20260927 15:19:14.577351 24977 raft_consensus.cc:504] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.577714 24977 leader_election.cc:288] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 election: Requested vote from peers c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709), 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813)
I20260927 15:19:14.578029 24914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661a7931c91f43e1997e4d7c6c7b98a9" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20c9518c859415f8d6da1af1f90f600"
I20260927 15:19:14.578028 24644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661a7931c91f43e1997e4d7c6c7b98a9" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73835b012405447faba430ac2b055bb0"
I20260927 15:19:14.578131 24914 raft_consensus.cc:3037] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:14.578142 24644 raft_consensus.cc:3037] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:14.579079 24914 raft_consensus.cc:2454] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 1.
I20260927 15:19:14.579231 24644 raft_consensus.cc:2454] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be118265b764a91a68ab3a3d23abb88 in term 1.
I20260927 15:19:14.579461 24715 leader_election.cc:302] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be118265b764a91a68ab3a3d23abb88, c20c9518c859415f8d6da1af1f90f600; no voters:
I20260927 15:19:14.579634 24977 raft_consensus.cc:2781] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:14.579850 24977 raft_consensus.cc:686] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 LEADER]: Becoming Leader. State: Replica: 2be118265b764a91a68ab3a3d23abb88, State: Running, Role: LEADER
I20260927 15:19:14.580183 24977 consensus_queue.cc:237] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:14.583182 24502 catalog_manager.cc:5495] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2be118265b764a91a68ab3a3d23abb88 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "2be118265b764a91a68ab3a3d23abb88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } health_report { overall_health: UNKNOWN } } }
W20260927 15:19:14.585034 24691 tablet.cc:2369] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:14.601851 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.598773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.9.242.130:40485)
0927 15:19:14.598885 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:14.598889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:14.598901 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:19:14.599005 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:14.599007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:14.599061 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:14.599142 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:14.599307 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.599789 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.600520 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:14.600722 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:14.600786 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:14.601076 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:14.601498 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:14.601615 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:14.601643 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 15:19:14.601740 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:19:14.607633 24914 raft_consensus.cc:1261] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Refusing update from remote peer 2be118265b764a91a68ab3a3d23abb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:14.607632 24644 raft_consensus.cc:1261] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Refusing update from remote peer 2be118265b764a91a68ab3a3d23abb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:14.608347 24977 consensus_queue.cc:1035] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:14.608553 24980 consensus_queue.cc:1035] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:14.624317 24501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:14.624475 24501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:14.626747 24501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Sending DeleteTablet for 3 replicas of tablet 661a7931c91f43e1997e4d7c6c7b98a9
I20260927 15:19:14.627498 24762 tablet_service.cc:1514] Processing DeleteTablet for tablet 661a7931c91f43e1997e4d7c6c7b98a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:14 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:53262
I20260927 15:19:14.627586 24894 tablet_service.cc:1514] Processing DeleteTablet for tablet 661a7931c91f43e1997e4d7c6c7b98a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:14 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:36318
I20260927 15:19:14.627699 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 24540
I20260927 15:19:14.628340 24993 tablet_replica.cc:330] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: stopping tablet replica
I20260927 15:19:14.628674 24993 raft_consensus.cc:2229] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:14.629020 24993 raft_consensus.cc:2258] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:14.630692 24993 ts_tablet_manager.cc:1905] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:14.633486 24993 ts_tablet_manager.cc:1918] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 15:19:14.633496 24486 connection.cc:535] client connection to 127.9.242.129:36813 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:19:14.633594 24993 log.cc:1196] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/661a7931c91f43e1997e4d7c6c7b98a9
W20260927 15:19:14.633632 24486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:19:14.633944 24993 ts_tablet_manager.cc:1939] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Deleting consensus metadata
I20260927 15:19:14.634752 24992 tablet_replica.cc:330] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: stopping tablet replica
I20260927 15:19:14.635103 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.634741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.0.0.1:57758)
0927 15:19:14.634949 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.635016 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:36813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 15:19:14.635077 24992 raft_consensus.cc:2229] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:14.635501 24992 raft_consensus.cc:2258] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 15:19:14.635258 24486 catalog_manager.cc:4584] TS 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 661a7931c91f43e1997e4d7c6c7b98a9: Network error: Client connection negotiation failed: client connection to 127.9.242.129:36813: connect: Connection refused (error 111)
I20260927 15:19:14.635833 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 24694
I20260927 15:19:14.635964 24486 catalog_manager.cc:4841] TS c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709): tablet 661a7931c91f43e1997e4d7c6c7b98a9 (table test-table [id=59015dede3c14e4a9dbce1514baf79fe]) successfully deleted
I20260927 15:19:14.640798 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 24831
I20260927 15:19:14.641001 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:14.640727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40485 (local address 127.0.0.1:53280)
0927 15:19:14.640875 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:14.640924 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.130:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 15:19:14.645361 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 24469
I20260927 15:19:14.650490 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39655
--webserver_interface=127.9.242.190
--webserver_port=44695
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:39655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:14.759104 24995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:14.759369 24995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:14.759408 24995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:14.763391 24995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:14.763453 24995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:14.763509 24995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:14.763533 24995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:14.763564 24995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:14.768852 24995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:39655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:14.769857 24995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:14.770830 24995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:14.778930 25000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.778966 25003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:14.778934 24995 server_base.cc:1034] running on GCE node
W20260927 15:19:14.778983 25001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:14.779789 24995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:14.780889 24995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:14.782056 24995 hybrid_clock.cc:648] HybridClock initialized: now 1790522354782026 us; error 44 us; skew 500 ppm
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:14.784658 24995 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:14.785588 24995 webserver.cc:460] Webserver started at http://127.9.242.190:44695/ using document root <none> and password file <none>
I20260927 15:19:14.786175 24995 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:14.786252 24995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:14.790203 24995 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:19:14.792337 25010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.793736 24995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:14.793849 24995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "ebe9ef9363d5425c8abc8e9a51743ac7"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "26e29f59816c46cd3eee67532ddcc215"
server_key_iv: "4924dd498bf452775d2fd6c5784ff1b5"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.794425 24995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:14.806169 24995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:14.809062 24995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:14.809245 24995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:14.815908 24995 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:39655
I20260927 15:19:14.815959 25062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:39655 every 8 connection(s)
I20260927 15:19:14.816934 24995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:14.821009 25063 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:19:14.822650 25063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Bootstrap starting.
I20260927 15:19:14.826125 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 24995
I20260927 15:19:14.826581 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:36813
--local_ip_for_outbound_sockets=127.9.242.129
--tserver_master_addrs=127.9.242.190:39655
--webserver_port=44557
--webserver_interface=127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:14.840351 25063 log.cc:824] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:14.851580 25063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:14.851943 25063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7: Bootstrap complete.
I20260927 15:19:14.855955 25063 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:14.856369 25063 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe9ef9363d5425c8abc8e9a51743ac7, State: Initialized, Role: FOLLOWER
I20260927 15:19:14.856830 25063 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:14.856943 25063 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:14.857003 25063 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:14.857131 25063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:19:14.859100 25063 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:14.859416 25063 leader_election.cc:302] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe9ef9363d5425c8abc8e9a51743ac7; no voters:
I20260927 15:19:14.859699 25063 leader_election.cc:288] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:19:14.859804 25067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:19:14.859985 25067 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [term 3 LEADER]: Becoming Leader. State: Replica: ebe9ef9363d5425c8abc8e9a51743ac7, State: Running, Role: LEADER
I20260927 15:19:14.860267 25067 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } }
I20260927 15:19:14.860911 25063 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:14.861313 25067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe9ef9363d5425c8abc8e9a51743ac7. Latest consensus state: current_term: 3 leader_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } } }
I20260927 15:19:14.861416 25067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:14.861783 25071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:14.861760 25068 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe9ef9363d5425c8abc8e9a51743ac7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39655 } } }
I20260927 15:19:14.861963 25068 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe9ef9363d5425c8abc8e9a51743ac7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:14.866783 25071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:14.867421 25071 catalog_manager.cc:1259] Loaded cluster ID: 18af19be2ce542b9a563587e558f7374
I20260927 15:19:14.867482 25071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 15:19:14.868435 25071 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcc0803c9ad google::LogMessage::Fail() at ??:0
@ 0x7fcc0803f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcc0803c4de google::LogMessage::Flush() at ??:0
@ 0x7fcc0803da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcc1297bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fcc1297cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcc12979579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fcc129bd640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcc0a28e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcc0a28ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fcc0a2903f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcc0a27f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcc0b8c36db start_thread at ??:0
@ 0x7fcc06b3a61f clone at ??:0
W20260927 15:19:14.949535 25065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:14.949838 25065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:14.949911 25065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:14.953902 25065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:14.953953 25065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:14.954025 25065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:19:14.959206 25065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:36813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=44557
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:14.960192 25065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:14.961155 25065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:14.970510 25107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.970497 25105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:14.970570 25104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:14.971279 25065 server_base.cc:1034] running on GCE node
I20260927 15:19:14.971765 25065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:14.972584 25065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:14.973739 25065 hybrid_clock.cc:648] HybridClock initialized: now 1790522354973704 us; error 43 us; skew 500 ppm
Sep 27 15:19:14 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:14.976979 25065 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:19:14.978152 25065 webserver.cc:460] Webserver started at http://127.9.242.129:44557/ using document root <none> and password file <none>
I20260927 15:19:14.978864 25065 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:14.978957 25065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:14.984030 25065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:19:14.986800 25114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:14.988297 25065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:14.988415 25065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "73835b012405447faba430ac2b055bb0"
format_stamp: "Formatted at 2026-09-27 15:19:12 on dist-test-slave-m4lt"
server_key: "5d70b5f9034203b2d936c05b3c11dbfb"
server_key_iv: "97348ebd7907c271fac5fb9bf9354d85"
server_key_version: "encryptionkey@0"
I20260927 15:19:14.989123 25065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:15.013701 25065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:15.017741 25065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:15.017948 25065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:15.018689 25065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:15.020187 25121 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:19:15.024379 25065 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:19:15.024462 25065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 15:19:15.024529 25065 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:19:15.026799 25065 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:19:15.026861 25065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 15:19:15.027202 25121 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Bootstrap starting.
I20260927 15:19:15.037253 25065 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:36813
I20260927 15:19:15.037521 25228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:36813 every 8 connection(s)
I20260927 15:19:15.038491 25065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:15.043861 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 25065
I20260927 15:19:15.044329 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:40485
--local_ip_for_outbound_sockets=127.9.242.130
--tserver_master_addrs=127.9.242.190:39655
--webserver_port=41369
--webserver_interface=127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:15.047065 25121 log.cc:824] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:15.055868 25121 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:15.056288 25121 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Bootstrap complete.
I20260927 15:19:15.057554 25121 ts_tablet_manager.cc:1397] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.004s
I20260927 15:19:15.060735 25121 raft_consensus.cc:348] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:15.061276 25121 raft_consensus.cc:729] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73835b012405447faba430ac2b055bb0, State: Initialized, Role: FOLLOWER
I20260927 15:19:15.061836 25121 consensus_queue.cc:260] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:15.062885 25121 ts_tablet_manager.cc:1428] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260927 15:19:15.086793 25231 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:19:15.040782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:41707)
0927 15:19:15.041439 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.041498 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:15.042651 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:15.085759 (+ 43108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":129,"threads_started":1}
W20260927 15:19:15.087361 25229 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:39655 (0 consecutive failures): Network error: Failed to ping master at 127.9.242.190:39655: Client connection negotiation failed: client connection to 127.9.242.190:39655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:19:15.088090 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.087810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:36109)
0927 15:19:15.087946 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.088026 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 15:19:15.088727 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.088520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:50101)
0927 15:19:15.088624 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.088671 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 15:19:15.088934 25229 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:19:15.154989 25232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:15.155260 25232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:15.155305 25232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:15.159346 25232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:15.159405 25232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:15.159469 25232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:19:15.164925 25232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:40485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=41369
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:15.165998 25232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:15.166954 25232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:15.176084 25239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:15.176110 25242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:15.176155 25232 server_base.cc:1034] running on GCE node
W20260927 15:19:15.176105 25240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:15.176697 25232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:15.177316 25232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:15.178462 25232 hybrid_clock.cc:648] HybridClock initialized: now 1790522355178433 us; error 42 us; skew 500 ppm
Sep 27 15:19:15 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:15.180915 25232 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:19:15.181828 25232 webserver.cc:460] Webserver started at http://127.9.242.130:41369/ using document root <none> and password file <none>
I20260927 15:19:15.182348 25232 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:15.182410 25232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:15.186224 25232 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 15:19:15.188282 25249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.189790 25232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:15.189908 25232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "2be118265b764a91a68ab3a3d23abb88"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "65fd1bfee6f98207948e30c992d3f0cb"
server_key_iv: "afa8b64bd2d3508e7b7791b9e1383860"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.190462 25232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:15.216948 25232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:15.220374 25232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:15.220559 25232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:15.221148 25232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:15.222282 25256 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:19:15.226518 25232 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:19:15.226594 25232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 15:19:15.226665 25232 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:19:15.228407 25232 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:19:15.228456 25232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:19:15.228729 25256 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Bootstrap starting.
I20260927 15:19:15.237763 25232 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:40485
I20260927 15:19:15.237871 25363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:40485 every 8 connection(s)
I20260927 15:19:15.238706 25232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:15.241741 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 25232
I20260927 15:19:15.242115 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:43709
--local_ip_for_outbound_sockets=127.9.242.131
--tserver_master_addrs=127.9.242.190:39655
--webserver_port=33285
--webserver_interface=127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:15.242528 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.240799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:33583)
0927 15:19:15.241341 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.241521 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
W20260927 15:19:15.243148 25364 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:39655 (0 consecutive failures): Network error: Failed to ping master at 127.9.242.190:39655: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
I20260927 15:19:15.243841 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.243583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:39259)
0927 15:19:15.243715 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.243775 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:19:15.244583 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.244369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:43725)
0927 15:19:15.244478 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.244526 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 15:19:15.244729 25364 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:19:15.247433 25256 log.cc:824] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:15.253966 25256 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:15.254276 25256 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Bootstrap complete.
I20260927 15:19:15.255164 25256 ts_tablet_manager.cc:1397] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20260927 15:19:15.257711 25256 raft_consensus.cc:348] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:15.258149 25256 raft_consensus.cc:729] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be118265b764a91a68ab3a3d23abb88, State: Initialized, Role: FOLLOWER
I20260927 15:19:15.258575 25256 consensus_queue.cc:260] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:15.259541 25256 ts_tablet_manager.cc:1428] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 15:19:15.259778 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.259510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:60023)
0927 15:19:15.259635 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.259700 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:19:15.260378 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.260186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:56983)
0927 15:19:15.260278 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.260324 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:19:15.351372 25367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:15.351640 25367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:15.351682 25367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:15.355659 25367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:15.355716 25367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:15.355774 25367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:19:15.360910 25367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:43709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=33285
--tserver_master_addrs=127.9.242.190:39655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:15.361936 25367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:15.362887 25367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:15.372263 25374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:15.372293 25375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:15.372273 25377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:15.372815 25367 server_base.cc:1034] running on GCE node
I20260927 15:19:15.373208 25367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:15.373867 25367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:15.375020 25367 hybrid_clock.cc:648] HybridClock initialized: now 1790522355374992 us; error 41 us; skew 500 ppm
Sep 27 15:19:15 dist-test-slave-m4lt krb5kdc[23921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:15.377707 25367 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:19:15.378557 25367 webserver.cc:460] Webserver started at http://127.9.242.131:33285/ using document root <none> and password file <none>
I20260927 15:19:15.379086 25367 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:15.379163 25367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:15.383025 25367 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 15:19:15.385047 25384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.386315 25367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:15.386425 25367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "c20c9518c859415f8d6da1af1f90f600"
format_stamp: "Formatted at 2026-09-27 15:19:13 on dist-test-slave-m4lt"
server_key: "dc5aeb53ff19ca21305604c927dbb357"
server_key_iv: "b62c47387254599a79e9bcae54fd0129"
server_key_version: "encryptionkey@0"
I20260927 15:19:15.386941 25367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:15.411778 25367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:15.414835 25367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:15.415005 25367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:15.415568 25367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:15.416354 25367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:15.416400 25367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.416451 25367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:15.416481 25367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:15.424477 25367 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:43709
I20260927 15:19:15.424511 25497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:43709 every 8 connection(s)
I20260927 15:19:15.425416 25367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:15.429189 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.427529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:60839)
0927 15:19:15.428014 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.428449 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":109,"threads_started":1}
W20260927 15:19:15.429684 25498 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:39655 (0 consecutive failures): Network error: Failed to ping master at 127.9.242.190:39655: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
I20260927 15:19:15.429924 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 25367
I20260927 15:19:15.430270 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.430057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:42883)
0927 15:19:15.430174 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.430220 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 15:19:15.430882 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:15.430646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:58341)
0927 15:19:15.430759 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:15.430810 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 15:19:15.431099 25498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:19:16.089957 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.089481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:51623)
0927 15:19:16.089778 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.089842 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":71,"threads_started":1}
I20260927 15:19:16.261528 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.261140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:53459)
0927 15:19:16.261367 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.261436 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260927 15:19:16.432111 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.431700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:55461)
0927 15:19:16.431946 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.432007 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260927 15:19:16.723789 25505 raft_consensus.cc:482] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:16.723978 25505 raft_consensus.cc:504] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:16.725040 25505 leader_election.cc:288] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709), 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813)
I20260927 15:19:16.725369 25506 raft_consensus.cc:482] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:16.725543 25506 raft_consensus.cc:504] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:16.727124 25506 leader_election.cc:288] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709), 2be118265b764a91a68ab3a3d23abb88 (127.9.242.130:40485)
Sep 27 15:19:16 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.130@KRBTEST.COM
Sep 27 15:19:16 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522354, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.131@KRBTEST.COM
Sep 27 15:19:16 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.131@KRBTEST.COM
Sep 27 15:19:16 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:19:16.740336 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.727009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40485 (local address 127.9.242.129:34523)
0927 15:19:16.727804 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.727816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:16.727884 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:16.728350 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:16.728356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:16.728901 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:16.729362 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.729378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730864 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.730868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.731620 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.731630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.732350 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.732353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.732521 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.733226 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:16.733248 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:16.735120 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:16.737498 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.737504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.737519 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.737864 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.738248 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.738250 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.738252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.738451 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.739257 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:16.739264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:16.739529 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 15:19:16.739835 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 15:19:16.740148 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":561,"spinlock_wait_cycles":1920,"thread_start_us":55,"threads_started":1}
I20260927 15:19:16.740566 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.725279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43709 (local address 127.9.242.130:54567)
0927 15:19:16.725414 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.725447 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:16.725940 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:16.727828 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:16.727832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:16.728138 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:16.728801 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.728823 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730673 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.730676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.731802 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.731812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.733150 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.733154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.733252 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.733715 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:16.733735 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:16.735986 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:16.738703 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.738712 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.738718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.738988 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.739301 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.739304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.739306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.739392 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.739827 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:16.739832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:16.739964 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 15:19:16.740109 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 15:19:16.740387 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":20}
I20260927 15:19:16.740729 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.727008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:43709 (local address 127.9.242.129:34223)
0927 15:19:16.727608 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.727622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:16.727704 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:16.728352 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:16.728357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:16.728901 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:16.729362 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.729377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730673 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.730676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.731687 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.731697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.732899 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.732903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.732985 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.733374 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:16.733390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:16.735350 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:16.738245 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.738248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.738250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.738508 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.738886 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.738889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.738891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.738970 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.739701 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:16.739704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:16.739785 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 15:19:16.739869 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:19:16.740582 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":648,"spinlock_wait_cycles":6272,"thread_start_us":69,"threads_started":1}
I20260927 15:19:16.741117 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.725454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:54567 (local address 127.9.242.131:43709)
0927 15:19:16.726230 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.726253 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.726299 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:19:16.727088 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.727096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.727556 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.728304 (+ 748us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.728985 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730529 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.732749 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.733053 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.733142 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.736191 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.736195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.736198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.736618 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.738586 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.739111 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.739114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.739116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.739198 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.739497 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.739500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.739502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.739709 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.740733 (+ 1024us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.740806 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.740910 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":653,"thread_start_us":99,"threads_started":1}
I20260927 15:19:16.741151 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.727323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:34223 (local address 127.9.242.131:43709)
0927 15:19:16.728111 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.728121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.728133 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:19:16.728163 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.728165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.728203 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.728475 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.729505 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730529 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.732102 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.732800 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.732935 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.735521 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.735526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.735539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.735566 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.738127 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.738635 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.738640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.738646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.738781 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.739064 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.739067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.739069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.739534 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.739657 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.740091 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.740364 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":720,"thread_start_us":105,"threads_started":1}
I20260927 15:19:16.742380 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.727324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:34523 (local address 127.9.242.130:40485)
0927 15:19:16.727918 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.727924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.727943 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:19:16.727981 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.727985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.728149 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.728553 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.729520 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730748 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.731743 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.732238 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.732390 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.735285 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.735292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.735308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.735342 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.737343 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.738000 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.738007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.738014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.738158 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.738558 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.738562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.738564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.739132 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.739653 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.740935 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.741075 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":59,"threads_started":1}
I20260927 15:19:16.742686 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.725556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.9.242.130:38869)
0927 15:19:16.726251 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.726264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:16.726352 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:16.727581 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:16.727591 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:16.728123 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:16.728935 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.728942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730700 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.730703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.732588 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:16.732598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.733970 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.733974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:16.734054 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.734509 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:16.734520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:16.737920 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:16.739955 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.739957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.739959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.740179 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.741475 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:16.741477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:16.741479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:16.741551 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:16.741982 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:16.741985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:16.742069 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 15:19:16.742178 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 15:19:16.742303 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":1088,"thread_start_us":89,"threads_started":1}
I20260927 15:19:16.743678 25318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661a7931c91f43e1997e4d7c6c7b98a9" candidate_uuid: "73835b012405447faba430ac2b055bb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2be118265b764a91a68ab3a3d23abb88" is_pre_election: true
W20260927 15:19:16.743719 25115 leader_election.cc:341] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709): Not found: Tablet not found: 661a7931c91f43e1997e4d7c6c7b98a9
W20260927 15:19:16.743764 25250 leader_election.cc:341] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709): Not found: Tablet not found: 661a7931c91f43e1997e4d7c6c7b98a9
I20260927 15:19:16.743888 25318 raft_consensus.cc:2454] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73835b012405447faba430ac2b055bb0 in term 1.
I20260927 15:19:16.744532 25118 leader_election.cc:302] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be118265b764a91a68ab3a3d23abb88, 73835b012405447faba430ac2b055bb0; no voters: c20c9518c859415f8d6da1af1f90f600
I20260927 15:19:16.744797 25506 raft_consensus.cc:2781] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 15:19:16.744844 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.725941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:38869 (local address 127.9.242.129:36813)
0927 15:19:16.726348 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:16.726354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:16.726378 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:19:16.726518 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:16.726524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:16.726785 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:16.726956 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:16.729122 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.730566 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.732706 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:16.733867 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:16.733970 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:16.738066 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:16.738086 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:16.738098 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:16.738124 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:16.739857 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.740283 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.740288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.740289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.740370 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:16.741648 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:16.741670 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:16.741672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:16.741885 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:16.741991 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:16.744552 (+ 2561us) server_negotiation.cc:299] Negotiation successful
0927 15:19:16.744705 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":315,"thread_start_us":55,"threads_started":1}
I20260927 15:19:16.744868 25506 raft_consensus.cc:482] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:16.744915 25506 raft_consensus.cc:3037] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:19:16.745568 25183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661a7931c91f43e1997e4d7c6c7b98a9" candidate_uuid: "2be118265b764a91a68ab3a3d23abb88" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "73835b012405447faba430ac2b055bb0" is_pre_election: true
I20260927 15:19:16.747529 25506 raft_consensus.cc:504] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:16.747776 25183 raft_consensus.cc:2379] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2be118265b764a91a68ab3a3d23abb88 in current term 2: Already voted for candidate 73835b012405447faba430ac2b055bb0 in this term.
I20260927 15:19:16.747892 25506 leader_election.cc:288] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [CANDIDATE]: Term 2 election: Requested vote from peers c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709), 2be118265b764a91a68ab3a3d23abb88 (127.9.242.130:40485)
I20260927 15:19:16.748282 25318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "661a7931c91f43e1997e4d7c6c7b98a9" candidate_uuid: "73835b012405447faba430ac2b055bb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2be118265b764a91a68ab3a3d23abb88"
I20260927 15:19:16.748414 25318 raft_consensus.cc:3037] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 1 FOLLOWER]: Advancing to term 2
W20260927 15:19:16.748519 25115 leader_election.cc:341] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709): Not found: Tablet not found: 661a7931c91f43e1997e4d7c6c7b98a9
I20260927 15:19:16.749029 25250 leader_election.cc:302] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2be118265b764a91a68ab3a3d23abb88; no voters: 73835b012405447faba430ac2b055bb0, c20c9518c859415f8d6da1af1f90f600
I20260927 15:19:16.750581 25318 raft_consensus.cc:2454] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73835b012405447faba430ac2b055bb0 in term 2.
I20260927 15:19:16.750682 25505 raft_consensus.cc:2726] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:19:16.750931 25118 leader_election.cc:302] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2be118265b764a91a68ab3a3d23abb88, 73835b012405447faba430ac2b055bb0; no voters: c20c9518c859415f8d6da1af1f90f600
I20260927 15:19:16.751179 25506 raft_consensus.cc:2781] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:19:16.751356 25506 raft_consensus.cc:686] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [term 2 LEADER]: Becoming Leader. State: Replica: 73835b012405447faba430ac2b055bb0, State: Running, Role: LEADER
I20260927 15:19:16.751652 25506 consensus_queue.cc:237] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:16.752066 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:16.751829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:59839)
0927 15:19:16.751950 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:16.752003 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 15:19:17.175073 25318 raft_consensus.cc:1261] T 661a7931c91f43e1997e4d7c6c7b98a9 P 2be118265b764a91a68ab3a3d23abb88 [term 2 FOLLOWER]: Refusing update from remote peer 73835b012405447faba430ac2b055bb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 15:19:17.175791 25506 consensus_queue.cc:1035] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 15:19:17.177477 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.177221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:58707)
0927 15:19:17.177343 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:17.177403 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 15:19:17.180009 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.179754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:38565)
0927 15:19:17.179894 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:17.179938 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 15:19:17.180284 25115 consensus_peers.cc:487] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0 -> Peer c20c9518c859415f8d6da1af1f90f600 (127.9.242.131:43709): Couldn't send request to peer c20c9518c859415f8d6da1af1f90f600. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 661a7931c91f43e1997e4d7c6c7b98a9. This is attempt 1: this message will repeat every 5th retry.
I20260927 15:19:17.180559 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.180272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:45853)
0927 15:19:17.180456 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:17.180501 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 15:19:17.433279 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.432834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:42879)
0927 15:19:17.433109 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:17.433171 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260927 15:19:17.717909 25521 ts_tablet_manager.cc:927] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Initiating tablet copy from peer 73835b012405447faba430ac2b055bb0 (127.9.242.129:36813)
I20260927 15:19:17.718514 25521 tablet_copy_client.cc:316] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: tablet copy: Beginning tablet copy session from remote peer at address 127.9.242.129:36813
Sep 27 15:19:17 dist-test-slave-m4lt krb5kdc[23921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522355, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:19:17.733006 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.719185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:42139 (local address 127.9.242.129:36813)
0927 15:19:17.719480 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:17.719485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:17.719498 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:19:17.719535 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:17.719538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:17.719578 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:17.719654 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:17.720437 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:17.720981 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:17.721904 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:17.722124 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:17.722200 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:17.729408 (+ 7208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:17.729412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:17.729414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:17.729443 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:17.731135 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:17.731745 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:17.731748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:17.731749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:17.731847 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:17.732107 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:17.732109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:17.732111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:17.732311 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:17.732387 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:17.732778 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 15:19:17.732881 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260927 15:19:17.733009 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.719026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36813 (local address 127.9.242.131:42139)
0927 15:19:17.719279 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:17.719305 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:17.719447 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:17.719716 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:17.719723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:17.720211 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:17.720329 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:17.720335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:17.721090 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:17.721093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:17.721812 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:17.721822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:17.722242 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:17.722245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:17.722332 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:17.727679 (+ 5347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:17.727694 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:17.729209 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:17.731283 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:17.731293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:17.731299 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:17.731547 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:17.731934 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:17.731937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:17.731938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:17.732022 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:17.732405 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:17.732411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:17.732531 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 15:19:17.732721 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 15:19:17.732862 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260927 15:19:17.733681 25203 tablet_copy_service.cc:140] P 73835b012405447faba430ac2b055bb0: Received BeginTabletCopySession request for tablet 661a7931c91f43e1997e4d7c6c7b98a9 from peer c20c9518c859415f8d6da1af1f90f600 ({username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:42139)
I20260927 15:19:17.733831 25203 tablet_copy_service.cc:161] P 73835b012405447faba430ac2b055bb0: Beginning new tablet copy session on tablet 661a7931c91f43e1997e4d7c6c7b98a9 from peer c20c9518c859415f8d6da1af1f90f600 at {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:42139: session id = c20c9518c859415f8d6da1af1f90f600-661a7931c91f43e1997e4d7c6c7b98a9
I20260927 15:19:17.734992 25203 tablet_copy_source_session.cc:205] T 661a7931c91f43e1997e4d7c6c7b98a9 P 73835b012405447faba430ac2b055bb0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 15:19:17.738757 25521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 661a7931c91f43e1997e4d7c6c7b98a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:17.745460 25521 tablet_copy_client.cc:791] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: tablet copy: Starting download of 0 data blocks...
I20260927 15:19:17.745637 25521 tablet_copy_client.cc:655] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: tablet copy: Starting download of 1 WAL segments...
I20260927 15:19:17.747394 25521 tablet_copy_client.cc:531] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 15:19:17.749097 25521 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Bootstrap starting.
I20260927 15:19:17.754348 25521 log.cc:824] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:17.760560 25521 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:17.760830 25521 tablet_bootstrap.cc:492] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Bootstrap complete.
I20260927 15:19:17.761119 25521 ts_tablet_manager.cc:1397] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 15:19:17.763345 25521 raft_consensus.cc:348] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:17.763626 25521 raft_consensus.cc:729] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20c9518c859415f8d6da1af1f90f600, State: Initialized, Role: FOLLOWER
I20260927 15:19:17.764099 25521 consensus_queue.cc:260] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20c9518c859415f8d6da1af1f90f600" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 43709 } } peers { permanent_uuid: "2be118265b764a91a68ab3a3d23abb88" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40485 } } peers { permanent_uuid: "73835b012405447faba430ac2b055bb0" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36813 } }
I20260927 15:19:17.765091 25521 ts_tablet_manager.cc:1428] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260927 15:19:17.765280 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.765019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:44041)
0927 15:19:17.765141 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:17.765198 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 15:19:17.765789 25203 tablet_copy_service.cc:342] P 73835b012405447faba430ac2b055bb0: Request end of tablet copy session c20c9518c859415f8d6da1af1f90f600-661a7931c91f43e1997e4d7c6c7b98a9 received from {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:42139
I20260927 15:19:17.765908 25203 tablet_copy_service.cc:433] P 73835b012405447faba430ac2b055bb0: ending tablet copy session c20c9518c859415f8d6da1af1f90f600-661a7931c91f43e1997e4d7c6c7b98a9 on tablet 661a7931c91f43e1997e4d7c6c7b98a9 with peer c20c9518c859415f8d6da1af1f90f600
I20260927 15:19:17.765946 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:17.765735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:45855)
0927 15:19:17.765834 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:17.765885 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 15:19:18.125001 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:18.124636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:43303)
0927 15:19:18.124841 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:18.124899 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260927 15:19:18.125197 25452 raft_consensus.cc:1203] T 661a7931c91f43e1997e4d7c6c7b98a9 P c20c9518c859415f8d6da1af1f90f600 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 15:19:18.126107 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:18.125895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:36103)
0927 15:19:18.125987 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:18.126034 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 15:19:18.181552 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:18.181211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:32953)
0927 15:19:18.181412 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:18.181464 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260927 15:19:19.126189 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:19.125752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:34721)
0927 15:19:19.126018 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:19.126079 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
I20260927 15:19:19.127161 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:19.126779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:32919)
0927 15:19:19.127028 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:19.127082 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":59,"threads_started":1}
I20260927 15:19:19.182775 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:19.182314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:51873)
0927 15:19:19.182614 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:19.182678 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":69,"threads_started":1}
I20260927 15:19:20.127523 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.127110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:34013)
0927 15:19:20.127364 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.127426 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260927 15:19:20.128207 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.128009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:57123)
0927 15:19:20.128098 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.128145 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 15:19:20.128360 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.127855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:56019)
0927 15:19:20.128151 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.128223 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
I20260927 15:19:20.184018 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:20.183569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:50381)
0927 15:19:20.183858 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:20.183921 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":73,"threads_started":1}
I20260927 15:19:21.129462 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.129029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:38735)
0927 15:19:21.129299 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:21.129363 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
I20260927 15:19:21.129691 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.129233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:37291)
0927 15:19:21.129444 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:21.129497 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260927 15:19:21.185467 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:21.185019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:51423)
0927 15:19:21.185303 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:21.185368 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260927 15:19:22.130667 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:22.130264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:49811)
0927 15:19:22.130511 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:22.130570 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260927 15:19:22.130760 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:22.130312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:46717)
0927 15:19:22.130605 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:22.130664 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":54,"threads_started":1}
I20260927 15:19:22.186713 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:22.186297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:42831)
0927 15:19:22.186524 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:22.186581 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260927 15:19:23.131901 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:23.131510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:47409)
0927 15:19:23.131757 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:23.131811 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260927 15:19:23.131907 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:23.131488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:33531)
0927 15:19:23.131751 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:23.131809 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260927 15:19:23.187880 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:23.187439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:38645)
0927 15:19:23.187728 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:23.187785 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":81,"threads_started":1}
I20260927 15:19:24.133051 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.132674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:37809)
0927 15:19:24.132907 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:24.132961 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260927 15:19:24.133100 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.132722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:51897)
0927 15:19:24.132957 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:24.133011 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260927 15:19:24.189002 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:24.188606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:48151)
0927 15:19:24.188846 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:24.188909 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260927 15:19:25.134248 25548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.133842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:54495)
0927 15:19:25.134082 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.134138 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260927 15:19:25.134321 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.133961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:42767)
0927 15:19:25.134185 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.134235 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260927 15:19:25.190201 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:25.189792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:57781)
0927 15:19:25.190032 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:25.190096 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260927 15:19:26.135391 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.134980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:58451)
0927 15:19:26.135234 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.135298 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260927 15:19:26.135432 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.135071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:46447)
0927 15:19:26.135297 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.135349 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260927 15:19:26.191531 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:26.191085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:38985)
0927 15:19:26.191362 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:26.191431 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260927 15:19:27.136565 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:27.136174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:46203)
0927 15:19:27.136418 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:27.136473 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260927 15:19:27.136643 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:27.136224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:41185)
0927 15:19:27.136488 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:27.136551 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260927 15:19:27.192824 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:27.192261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:39693)
0927 15:19:27.192662 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:27.192727 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
I20260927 15:19:28.137794 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:28.137337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:48185)
0927 15:19:28.137601 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:28.137683 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260927 15:19:28.137845 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:28.137340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:60767)
0927 15:19:28.137629 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:28.137760 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
I20260927 15:19:28.194023 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:28.193583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:48855)
0927 15:19:28.193858 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:28.193931 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":119,"threads_started":1}
I20260927 15:19:29.138962 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.138560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:33863)
0927 15:19:29.138814 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:29.138867 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260927 15:19:29.138983 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.138620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:46387)
0927 15:19:29.138847 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:29.138898 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":65,"threads_started":1}
I20260927 15:19:29.195094 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:29.194694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:42865)
0927 15:19:29.194945 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:29.194998 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
I20260927 15:19:30.140170 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:30.139779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.131:42121)
0927 15:19:30.140019 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:30.140077 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260927 15:19:30.140239 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:30.139866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.129:57033)
0927 15:19:30.140097 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:30.140150 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
I20260927 15:19:30.196264 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:30.195849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39655 (local address 127.9.242.130:51909)
0927 15:19:30.196097 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:30.196169 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:39655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260927 15:19:30.440052 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 25065
I20260927 15:19:30.444998 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 25232
I20260927 15:19:30.451068 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 25367
2026-09-27T15:19:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18968 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:19:30 dist-test-slave-m4lt krb5kdc[25568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:19:30 dist-test-slave-m4lt krb5kdc[25568](info): set up 2 sockets
Sep 27 15:19:30 dist-test-slave-m4lt krb5kdc[25568](info): commencing operation
krb5kdc: starting...
W20260927 15:19:31.417738 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:19:31 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:19:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:19:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:19:31.544621 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36981
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:36981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:31.652732 25581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:31.653023 25581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:31.653083 25581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:31.657307 25581 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:19:31.657371 25581 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:19:31.657392 25581 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:19:31.657409 25581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:31.657433 25581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:31.657457 25581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:31.657487 25581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:31.657524 25581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:31.662964 25581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:36981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:31.664039 25581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:31.665045 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:31.673344 25586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:31.673394 25581 server_base.cc:1034] running on GCE node
W20260927 15:19:31.673352 25587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:31.673380 25589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:31.674124 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:31.675165 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:31.676332 25581 hybrid_clock.cc:648] HybridClock initialized: now 1790522371676318 us; error 30 us; skew 500 ppm
Sep 27 15:19:31 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:31.679090 25581 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:31.679998 25581 webserver.cc:460] Webserver started at http://127.9.242.190:35443/ using document root <none> and password file <none>
I20260927 15:19:31.680606 25581 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:31.680681 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:31.680884 25581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:31.682938 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "a4f5d553952b46c4839c35abc15ee53f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "a4a8ae64d95bf2ce7133b8e48ba192ae"
server_key_iv: "85950a1bab3cad583a194aa73e394b8d"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.683451 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "a4f5d553952b46c4839c35abc15ee53f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "a4a8ae64d95bf2ce7133b8e48ba192ae"
server_key_iv: "85950a1bab3cad583a194aa73e394b8d"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.687069 25581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:19:31.689368 25596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:31.690647 25581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:31.690765 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "a4f5d553952b46c4839c35abc15ee53f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "a4a8ae64d95bf2ce7133b8e48ba192ae"
server_key_iv: "85950a1bab3cad583a194aa73e394b8d"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.690853 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:31.701388 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:31.704396 25581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:31.704579 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:31.711064 25581 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:36981
I20260927 15:19:31.711308 25648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:36981 every 8 connection(s)
I20260927 15:19:31.712014 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:31.714753 25649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:31.720101 25649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Bootstrap starting.
I20260927 15:19:31.720463 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 25581
I20260927 15:19:31.720575 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:19:31.720925 10186 external_mini_cluster.cc:1428] Setting key 8e82844ef371d8e45b1992cea18bb884
I20260927 15:19:31.722785 25649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:31.723577 25649 log.cc:824] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:31.725333 25649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: No bootstrap required, opened a new log
I20260927 15:19:31.729202 25649 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:31.729490 25649 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:31.729571 25649 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f5d553952b46c4839c35abc15ee53f, State: Initialized, Role: FOLLOWER
Sep 27 15:19:31 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:31.730043 25649 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:31.730190 25649 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:31.730284 25649 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:31.730376 25649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:31.731499 25649 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:31.731843 25649 leader_election.cc:302] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4f5d553952b46c4839c35abc15ee53f; no voters:
I20260927 15:19:31.732132 25649 leader_election.cc:288] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:19:31.732215 25654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:31.732388 25654 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 LEADER]: Becoming Leader. State: Replica: a4f5d553952b46c4839c35abc15ee53f, State: Running, Role: LEADER
I20260927 15:19:31.732729 25654 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:31.733412 25649 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:31.735320 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.722271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.9.242.190:36981)
0927 15:19:31.722713 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.722728 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.722765 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:19:31.723518 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.723537 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.723890 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.724207 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:31.724964 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.726021 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.726872 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.727365 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.727510 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.730322 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:31.730329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:31.730347 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:31.730382 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:31.732050 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.732524 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.732530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.732535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.732668 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.732912 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.732916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.732917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.733420 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:31.733603 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.734063 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.734363 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":132,"threads_started":1}
I20260927 15:19:31.735941 25655 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4f5d553952b46c4839c35abc15ee53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } } }
I20260927 15:19:31.736328 25655 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:31.736083 25656 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4f5d553952b46c4839c35abc15ee53f. Latest consensus state: current_term: 1 leader_uuid: "a4f5d553952b46c4839c35abc15ee53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } } }
I20260927 15:19:31.736644 25656 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: This master's current role is: LEADER
W20260927 15:19:31.738772 25669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:19:31.738842 25669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:19:31.738911 25670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:31.739585 25670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:31.744793 25670 catalog_manager.cc:1347] Generated new cluster ID: 17c733edf771470da9f8c28ba8eb17e5
I20260927 15:19:31.744849 25670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:31.757479 25670 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:19:31.758492 25670 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:31.768558 25670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Generated new TSK 0
I20260927 15:19:31.769280 25670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:19:31.801391 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36981
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:31.910593 25675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:31.910863 25675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:31.910907 25675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:31.915316 25675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:31.915381 25675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:31.915443 25675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:19:31.920815 25675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:31.921882 25675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:31.922923 25675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:31.932271 25681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:31.932320 25680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:31.932338 25683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:31.932459 25675 server_base.cc:1034] running on GCE node
I20260927 15:19:31.932963 25675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:31.933560 25675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:31.934729 25675 hybrid_clock.cc:648] HybridClock initialized: now 1790522371934705 us; error 31 us; skew 500 ppm
Sep 27 15:19:31 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:31.937538 25675 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:19:31.938462 25675 webserver.cc:460] Webserver started at http://127.9.242.129:43733/ using document root <none> and password file <none>
I20260927 15:19:31.938989 25675 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:31.939077 25675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:31.939265 25675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:31.941256 25675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "dc163206f2f54bba9601ed069055e309"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "3db71d0e34eba7cd53906579de2ed807"
server_key_iv: "a00a7a5381d0a31debe94efafcaae7a3"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.941825 25675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "dc163206f2f54bba9601ed069055e309"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "3db71d0e34eba7cd53906579de2ed807"
server_key_iv: "a00a7a5381d0a31debe94efafcaae7a3"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.945453 25675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:19:31.947919 25690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:31.949090 25675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:31.949225 25675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "dc163206f2f54bba9601ed069055e309"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "3db71d0e34eba7cd53906579de2ed807"
server_key_iv: "a00a7a5381d0a31debe94efafcaae7a3"
server_key_version: "encryptionkey@0"
I20260927 15:19:31.949322 25675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:31.962582 25675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:31.965636 25675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:31.965852 25675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:31.966421 25675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:31.967236 25675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:31.967281 25675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:31.967332 25675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:31.967358 25675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:31.975837 25675 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:35629
I20260927 15:19:31.975871 25803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:35629 every 8 connection(s)
I20260927 15:19:31.976681 25675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:31.977294 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 25675
I20260927 15:19:31.977413 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:19:31.977758 10186 external_mini_cluster.cc:1428] Setting key 179d37241ec18de779ba4f53f404f22d
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Sep 27 15:19:31 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522371, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:31.993964 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.978582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:39755 (local address 127.9.242.190:36981)
0927 15:19:31.978707 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:31.978712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:31.979759 (+ 1047us) server_negotiation.cc:407] Connection header received
0927 15:19:31.981280 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:31.981284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:31.981337 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:31.981418 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:31.982674 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.983492 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.984640 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.984981 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.985088 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.990321 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:31.990324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:31.990326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:31.990353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:31.991661 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.992262 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.992266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.992267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.992355 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:31.992707 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:31.992709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:31.992710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:31.992896 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:31.992984 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:31.993716 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 15:19:31.993840 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:19:31.995173 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:31.978974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:39755)
0927 15:19:31.979525 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:31.979584 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:31.981077 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:31.981567 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:31.981574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:31.982022 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:31.982489 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.982502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.983647 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.983650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.984511 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:31.984520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:31.985143 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:31.985146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:31.985296 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:31.985912 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:31.985928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:31.990116 (+ 4188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:31.991780 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.991787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.991805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.992154 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.992445 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:31.992448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:31.992450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:31.992623 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:31.992998 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:31.993006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:31.993360 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 15:19:31.993617 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 15:19:31.994046 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":131,"threads_started":1}
I20260927 15:19:31.996645 25804 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36981
I20260927 15:19:31.996943 25804 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:31.997793 25804 heartbeater.cc:511] Master 127.9.242.190:36981 requested a full tablet report, sending...
I20260927 15:19:31.999420 25613 ts_manager.cc:194] Registered new tserver with Master: dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629)
I20260927 15:19:32.000931 25613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:39755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:19:32.006805 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36981
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:32.114342 25809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:32.114614 25809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:32.114657 25809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:32.118613 25809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:32.118673 25809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:32.118731 25809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:19:32.123891 25809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:32.124985 25809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:32.126025 25809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:32.135453 25815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:32.135473 25817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:32.135608 25814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:32.136106 25809 server_base.cc:1034] running on GCE node
I20260927 15:19:32.136493 25809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:32.137096 25809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:32.138247 25809 hybrid_clock.cc:648] HybridClock initialized: now 1790522372138229 us; error 33 us; skew 500 ppm
Sep 27 15:19:32 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:32.140854 25809 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:19:32.141760 25809 webserver.cc:460] Webserver started at http://127.9.242.130:44143/ using document root <none> and password file <none>
I20260927 15:19:32.142263 25809 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:32.142318 25809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:32.142468 25809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:32.144400 25809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "b6d2938b143c4a3da7a69abf30af0f15"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "a6f3f75e15aa839ec8faf45ffecb1e21"
server_key_iv: "e2b5b3f06b1849e6031679ae5f99fde6"
server_key_version: "encryptionkey@0"
I20260927 15:19:32.144903 25809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "b6d2938b143c4a3da7a69abf30af0f15"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "a6f3f75e15aa839ec8faf45ffecb1e21"
server_key_iv: "e2b5b3f06b1849e6031679ae5f99fde6"
server_key_version: "encryptionkey@0"
I20260927 15:19:32.148411 25809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 15:19:32.151026 25824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:32.152496 25809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:19:32.152606 25809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "b6d2938b143c4a3da7a69abf30af0f15"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "a6f3f75e15aa839ec8faf45ffecb1e21"
server_key_iv: "e2b5b3f06b1849e6031679ae5f99fde6"
server_key_version: "encryptionkey@0"
I20260927 15:19:32.152720 25809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:32.168087 25809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:32.171224 25809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:32.171392 25809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:32.171980 25809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:32.172796 25809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:32.172847 25809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:32.172902 25809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:32.172932 25809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:32.181540 25809 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:35841
I20260927 15:19:32.181560 25937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:35841 every 8 connection(s)
I20260927 15:19:32.182551 25809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:32.182776 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 25809
I20260927 15:19:32.182883 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:19:32.183221 10186 external_mini_cluster.cc:1428] Setting key 8cd9dd743f80a9b4e2d0de75d4e1340b
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Sep 27 15:19:32 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:32.200153 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.184389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:59059 (local address 127.9.242.190:36981)
0927 15:19:32.184502 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.184507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.185553 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 15:19:32.187145 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.187148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.187195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.187263 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:32.188840 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.189375 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.191062 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.191291 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.191370 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.196604 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:32.196607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:32.196610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:32.196636 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:32.198102 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:32.198606 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:32.198609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:32.198610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:32.198727 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:32.199042 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:32.199045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:32.199046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:32.199270 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:32.199360 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.199858 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.199985 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:19:32.200938 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.184917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:59059)
0927 15:19:32.185405 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.185447 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.186905 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.187480 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.187490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.187956 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:32.188682 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.188702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.189497 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.189501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.190948 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.190958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.191439 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.191444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.191671 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.192305 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:32.192326 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:32.196343 (+ 4017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:32.198218 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:32.198223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:32.198237 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:32.198488 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:32.198814 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:32.198817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:32.198819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:32.198949 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:32.199368 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:32.199375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:32.199618 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.199779 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.200104 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260927 15:19:32.202262 25938 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36981
I20260927 15:19:32.202487 25938 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:32.203003 25938 heartbeater.cc:511] Master 127.9.242.190:36981 requested a full tablet report, sending...
I20260927 15:19:32.204047 25613 ts_manager.cc:194] Registered new tserver with Master: b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841)
I20260927 15:19:32.204631 25613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:59059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:19:32.212142 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36981
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:32.319180 25943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:32.319428 25943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:32.319468 25943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:32.323544 25943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:32.323625 25943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:32.323686 25943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:19:32.329156 25943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:32.330227 25943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:32.331223 25943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:32.340811 25949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:32.340840 25951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:32.340860 25948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:32.341074 25943 server_base.cc:1034] running on GCE node
I20260927 15:19:32.341534 25943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:32.342216 25943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:32.343379 25943 hybrid_clock.cc:648] HybridClock initialized: now 1790522372343326 us; error 65 us; skew 500 ppm
Sep 27 15:19:32 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:32.346166 25943 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:19:32.347021 25943 webserver.cc:460] Webserver started at http://127.9.242.131:46589/ using document root <none> and password file <none>
I20260927 15:19:32.347569 25943 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:32.347671 25943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:32.347873 25943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:19:32.349908 25943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "72e1c6f5f12046d4867d24f854180773"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "5285dbf988a1294a6260464b1a2b074f"
server_key_iv: "59aa938e76297ce9e97e4e01c5d6eda3"
server_key_version: "encryptionkey@0"
I20260927 15:19:32.350435 25943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "72e1c6f5f12046d4867d24f854180773"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "5285dbf988a1294a6260464b1a2b074f"
server_key_iv: "59aa938e76297ce9e97e4e01c5d6eda3"
server_key_version: "encryptionkey@0"
I20260927 15:19:32.354213 25943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:19:32.356704 25958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:32.357931 25943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:19:32.358038 25943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "72e1c6f5f12046d4867d24f854180773"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "5285dbf988a1294a6260464b1a2b074f"
server_key_iv: "59aa938e76297ce9e97e4e01c5d6eda3"
server_key_version: "encryptionkey@0"
I20260927 15:19:32.358141 25943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:32.379740 25943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:32.382994 25943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:32.383183 25943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:32.383785 25943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:32.384642 25943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:32.384688 25943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:32.384730 25943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:32.384757 25943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:32.392977 25943 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:39599
I20260927 15:19:32.393002 26071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:39599 every 8 connection(s)
I20260927 15:19:32.393880 25943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:32.398257 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 25943
I20260927 15:19:32.398370 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:19:32.398679 10186 external_mini_cluster.cc:1428] Setting key 78aff1d3a28b0360484a6c6130012d65
Sep 27 15:19:32 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:32.406814 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.395663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:43671 (local address 127.9.242.190:36981)
0927 15:19:32.395785 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.395789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.396615 (+ 826us) server_negotiation.cc:407] Connection header received
0927 15:19:32.397891 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.397894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.397940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.398003 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:32.399420 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.399952 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.401052 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.401268 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.401352 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.403656 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:32.403659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:32.403661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:32.403684 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:32.404867 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:32.405330 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:32.405333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:32.405335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:32.405415 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:32.405745 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:32.405748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:32.405750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:32.405927 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:32.406010 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.406535 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.406656 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:19:32.407560 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.395994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:43671)
0927 15:19:32.396459 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.396504 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.397648 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.398217 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.398226 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.398714 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:32.399278 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.399292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.400072 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.400074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.400941 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.400950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.401401 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.401404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.401561 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.402135 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:32.402151 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:32.403496 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:32.404973 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:32.404978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:32.404993 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:32.405240 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:32.405494 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:32.405497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:32.405499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:32.405626 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:32.406031 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:32.406037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:32.406294 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.406452 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.406767 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260927 15:19:32.408542 26072 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36981
I20260927 15:19:32.408726 26072 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:32.409164 26072 heartbeater.cc:511] Master 127.9.242.190:36981 requested a full tablet report, sending...
I20260927 15:19:32.410097 25613 ts_manager.cc:194] Registered new tserver with Master: 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599)
I20260927 15:19:32.410657 25613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:43671
I20260927 15:19:32.412314 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:32.420922 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.414159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.9.242.190:36981)
0927 15:19:32.414282 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.414287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.414456 (+ 169us) server_negotiation.cc:407] Connection header received
0927 15:19:32.414569 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.414572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.414611 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.414706 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:32.415344 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.415839 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.416644 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.416848 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.416930 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.418199 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:32.418202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:32.418205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:32.418227 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:32.419457 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:32.419821 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:32.419825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:32.419826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:32.419896 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:32.420125 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:32.420128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:32.420130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:32.420331 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:32.420445 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.420666 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.420774 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:19:32.423749 25613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:32.426882 25613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:32.440866 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.434896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:35841 (local address 127.0.0.1:56282)
0927 15:19:32.436157 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.436212 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.436574 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.437089 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.437092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.437108 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.437177 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.437183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.438491 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.438495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.439568 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.439575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.440507 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.440510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.440589 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.440607 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.440679 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.440731 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260927 15:19:32.440980 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.435633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.9.242.130:35841)
0927 15:19:32.435895 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.435905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.436308 (+ 403us) server_negotiation.cc:407] Connection header received
0927 15:19:32.436757 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.436763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.436950 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.437114 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.437259 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.438344 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.439677 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.440209 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.440301 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.440452 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.440719 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.440821 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260927 15:19:32.441026 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.434991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55002 (local address 127.9.242.129:35629)
0927 15:19:32.435417 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.435428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.435452 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:19:32.435652 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.435658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.435869 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.436023 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.436548 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.437748 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.439370 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.440184 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.440305 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.440454 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.440803 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.440894 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":113,"threads_started":1}
I20260927 15:19:32.441159 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.435759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.9.242.131:39599)
0927 15:19:32.436503 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.436516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.436536 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:19:32.436605 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.436610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.436797 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.436923 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.437138 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.438045 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.439598 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.440370 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.440461 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.440578 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.440932 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.441025 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":148,"threads_started":1}
I20260927 15:19:32.441298 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.434836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35629 (local address 127.0.0.1:55002)
0927 15:19:32.435241 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.435293 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.435425 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.436217 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.436223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.436245 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.436413 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.436420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.437887 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.437890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.439241 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.439251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.440307 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.440310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.440408 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.440444 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.440548 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.440670 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":142,"thread_start_us":80,"threads_started":1}
I20260927 15:19:32.441344 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.435544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:39599 (local address 127.0.0.1:33942)
0927 15:19:32.436176 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.436188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.436268 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.436907 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.436910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.436927 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.437013 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.437019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.438165 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.438169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.439478 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.439486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.440541 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.440545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.440646 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.440670 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.440738 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.440806 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260927 15:19:32.443976 25738 tablet_service.cc:1467] Processing CreateTablet for tablet a82e76a9802741a5a526e55b96f0b209 (DEFAULT_TABLE table=test-table [id=640efb1f589245db957b040bf79eae16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:32.444191 25872 tablet_service.cc:1467] Processing CreateTablet for tablet a82e76a9802741a5a526e55b96f0b209 (DEFAULT_TABLE table=test-table [id=640efb1f589245db957b040bf79eae16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:32.444193 26006 tablet_service.cc:1467] Processing CreateTablet for tablet a82e76a9802741a5a526e55b96f0b209 (DEFAULT_TABLE table=test-table [id=640efb1f589245db957b040bf79eae16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:32.444940 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82e76a9802741a5a526e55b96f0b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:32.445184 25872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82e76a9802741a5a526e55b96f0b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:32.445191 26006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82e76a9802741a5a526e55b96f0b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:32.450932 26089 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Bootstrap starting.
I20260927 15:19:32.452786 26091 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Bootstrap starting.
I20260927 15:19:32.453241 26089 tablet_bootstrap.cc:654] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:32.454313 26089 log.cc:824] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:32.454655 26091 tablet_bootstrap.cc:654] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:32.455082 26090 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Bootstrap starting.
I20260927 15:19:32.456368 26091 log.cc:824] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:32.456629 26089 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: No bootstrap required, opened a new log
I20260927 15:19:32.457015 26089 ts_tablet_manager.cc:1397] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:19:32.457191 26090 tablet_bootstrap.cc:654] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:32.457839 26090 log.cc:824] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:32.459414 26090 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: No bootstrap required, opened a new log
I20260927 15:19:32.459587 26090 ts_tablet_manager.cc:1397] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20260927 15:19:32.459664 26091 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: No bootstrap required, opened a new log
I20260927 15:19:32.459838 26091 ts_tablet_manager.cc:1397] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 15:19:32.460480 26089 raft_consensus.cc:348] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.460742 26089 raft_consensus.cc:374] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:32.460821 26089 raft_consensus.cc:729] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc163206f2f54bba9601ed069055e309, State: Initialized, Role: FOLLOWER
I20260927 15:19:32.461279 26089 consensus_queue.cc:260] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.462103 25804 heartbeater.cc:503] Master 127.9.242.190:36981 was elected leader, sending a full tablet report...
I20260927 15:19:32.462342 26089 ts_tablet_manager.cc:1428] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:19:32.463042 26090 raft_consensus.cc:348] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.463304 26090 raft_consensus.cc:374] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:32.463359 26090 raft_consensus.cc:729] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d2938b143c4a3da7a69abf30af0f15, State: Initialized, Role: FOLLOWER
I20260927 15:19:32.463343 26091 raft_consensus.cc:348] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.463640 26091 raft_consensus.cc:374] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:32.463699 26091 raft_consensus.cc:729] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e1c6f5f12046d4867d24f854180773, State: Initialized, Role: FOLLOWER
I20260927 15:19:32.463799 26090 consensus_queue.cc:260] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.464067 26091 consensus_queue.cc:260] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.464942 26090 ts_tablet_manager.cc:1428] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 15:19:32.464983 26091 ts_tablet_manager.cc:1428] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260927 15:19:32.465292 26072 heartbeater.cc:503] Master 127.9.242.190:36981 was elected leader, sending a full tablet report...
I20260927 15:19:32.465579 25938 heartbeater.cc:503] Master 127.9.242.190:36981 was elected leader, sending a full tablet report...
I20260927 15:19:32.472447 26095 raft_consensus.cc:482] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:32.472605 26095 raft_consensus.cc:504] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.473577 26095 leader_election.cc:288] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841), 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599)
I20260927 15:19:32.475252 26097 raft_consensus.cc:482] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:32.475387 26097 raft_consensus.cc:504] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.476759 26097 leader_election.cc:288] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629), b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841)
W20260927 15:19:32.477759 25805 tablet.cc:2369] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:32.478228 26098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.473820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:39599 (local address 127.9.242.129:39071)
0927 15:19:32.474108 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.474120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.474217 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.474416 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.474418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.474431 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.474520 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.474526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.475446 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.475449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.476977 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.476984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.477789 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.477792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.477911 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.477935 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.478001 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.478081 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":279,"thread_start_us":108,"threads_started":1}
I20260927 15:19:32.478389 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.473792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:35841 (local address 127.9.242.129:57869)
0927 15:19:32.473942 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.473954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.474049 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.474298 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.474300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.474313 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.474417 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.474423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.475446 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.475449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.476799 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.476811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.477986 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.477989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.478107 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.478126 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.478192 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.478267 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":80}
I20260927 15:19:32.478711 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.473847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:57869 (local address 127.9.242.130:35841)
0927 15:19:32.473957 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.473961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.474095 (+ 134us) server_negotiation.cc:407] Connection header received
0927 15:19:32.474149 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.474152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.474202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.474282 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.474512 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.475313 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.477138 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.477770 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.477884 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.477930 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.478513 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.478592 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:19:32.479285 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.474053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:39071 (local address 127.9.242.131:39599)
0927 15:19:32.474157 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.474161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.474194 (+ 33us) server_negotiation.cc:407] Connection header received
0927 15:19:32.474294 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.474297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.474335 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.474402 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.474654 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.475345 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.477082 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.477548 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.477679 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.477732 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.478404 (+ 672us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.479179 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":693,"server-negotiator.queue_time_us":42}
I20260927 15:19:32.479254 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82e76a9802741a5a526e55b96f0b209" candidate_uuid: "dc163206f2f54bba9601ed069055e309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d2938b143c4a3da7a69abf30af0f15" is_pre_election: true
I20260927 15:19:32.479539 25892 raft_consensus.cc:2454] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc163206f2f54bba9601ed069055e309 in term 0.
I20260927 15:19:32.479960 25693 leader_election.cc:302] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6d2938b143c4a3da7a69abf30af0f15, dc163206f2f54bba9601ed069055e309; no voters:
I20260927 15:19:32.480049 26026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82e76a9802741a5a526e55b96f0b209" candidate_uuid: "dc163206f2f54bba9601ed069055e309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e1c6f5f12046d4867d24f854180773" is_pre_election: true
I20260927 15:19:32.480219 26026 raft_consensus.cc:2454] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc163206f2f54bba9601ed069055e309 in term 0.
I20260927 15:19:32.480242 26095 raft_consensus.cc:2781] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:32.480304 26095 raft_consensus.cc:482] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:32.480341 26095 raft_consensus.cc:3037] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:32.481452 26095 raft_consensus.cc:504] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.481680 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.476418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35629 (local address 127.9.242.131:34821)
0927 15:19:32.476953 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.476965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.477039 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.477332 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.477335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.477350 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.477449 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.477454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.478353 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.478356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.479490 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.479497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.481210 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.481214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.481363 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.481387 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.481465 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.481535 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":25}
I20260927 15:19:32.481694 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.476618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:34821 (local address 127.9.242.129:35629)
0927 15:19:32.476714 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.476718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.477020 (+ 302us) server_negotiation.cc:407] Connection header received
0927 15:19:32.477121 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.477132 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.477175 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.477234 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.477564 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.478238 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.480505 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.481085 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.481203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.481244 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.481485 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.481562 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:19:32.482157 25758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82e76a9802741a5a526e55b96f0b209" candidate_uuid: "72e1c6f5f12046d4867d24f854180773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc163206f2f54bba9601ed069055e309" is_pre_election: true
I20260927 15:19:32.482298 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82e76a9802741a5a526e55b96f0b209" candidate_uuid: "dc163206f2f54bba9601ed069055e309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d2938b143c4a3da7a69abf30af0f15"
I20260927 15:19:32.482367 25758 raft_consensus.cc:2379] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72e1c6f5f12046d4867d24f854180773 in current term 1: Already voted for candidate dc163206f2f54bba9601ed069055e309 in this term.
I20260927 15:19:32.482390 26026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82e76a9802741a5a526e55b96f0b209" candidate_uuid: "dc163206f2f54bba9601ed069055e309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e1c6f5f12046d4867d24f854180773"
I20260927 15:19:32.482406 25892 raft_consensus.cc:3037] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:32.482483 26026 raft_consensus.cc:3037] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:32.483435 26099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.476417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:35841 (local address 127.9.242.131:60997)
0927 15:19:32.478958 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.478971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:32.479068 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:32.479353 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:32.479355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:32.479371 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:32.479461 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.479466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.480351 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.480354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.481712 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:32.481723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.482672 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.482676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:32.482766 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.482788 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:19:32.482848 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:19:32.482913 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2480,"mutex_wait_us":55,"thread_start_us":108,"threads_started":1}
I20260927 15:19:32.483610 25892 raft_consensus.cc:2454] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc163206f2f54bba9601ed069055e309 in term 1.
I20260927 15:19:32.483610 26026 raft_consensus.cc:2454] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc163206f2f54bba9601ed069055e309 in term 1.
I20260927 15:19:32.483868 26095 leader_election.cc:288] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 1 election: Requested vote from peers b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841), 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599)
I20260927 15:19:32.483883 25693 leader_election.cc:302] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6d2938b143c4a3da7a69abf30af0f15, dc163206f2f54bba9601ed069055e309; no voters:
I20260927 15:19:32.484014 26095 raft_consensus.cc:2781] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:32.484098 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.476556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:60997 (local address 127.9.242.130:35841)
0927 15:19:32.477223 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.477231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.479033 (+ 1802us) server_negotiation.cc:407] Connection header received
0927 15:19:32.479150 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.479152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.479195 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.479265 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:32.479572 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.480242 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.481838 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.482448 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.482552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.482598 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.483170 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.483281 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":124,"threads_started":1}
I20260927 15:19:32.484202 26095 raft_consensus.cc:686] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 LEADER]: Becoming Leader. State: Replica: dc163206f2f54bba9601ed069055e309, State: Running, Role: LEADER
I20260927 15:19:32.484417 25892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82e76a9802741a5a526e55b96f0b209" candidate_uuid: "72e1c6f5f12046d4867d24f854180773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d2938b143c4a3da7a69abf30af0f15" is_pre_election: true
I20260927 15:19:32.484571 25892 raft_consensus.cc:2379] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72e1c6f5f12046d4867d24f854180773 in current term 1: Already voted for candidate dc163206f2f54bba9601ed069055e309 in this term.
I20260927 15:19:32.484535 26095 consensus_queue.cc:237] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:32.485105 25961 leader_election.cc:302] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72e1c6f5f12046d4867d24f854180773; no voters: b6d2938b143c4a3da7a69abf30af0f15, dc163206f2f54bba9601ed069055e309
I20260927 15:19:32.485356 26097 raft_consensus.cc:2726] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 15:19:32.487864 25607 catalog_manager.cc:5495] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc163206f2f54bba9601ed069055e309 (127.9.242.129). New cstate: current_term: 1 leader_uuid: "dc163206f2f54bba9601ed069055e309" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } health_report { overall_health: UNKNOWN } } }
I20260927 15:19:32.498081 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.494605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.9.242.129:35629)
0927 15:19:32.494750 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:32.494755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:32.494883 (+ 128us) server_negotiation.cc:407] Connection header received
0927 15:19:32.495035 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:32.495038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:32.495089 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:32.495170 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:32.495392 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.495880 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.496708 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:32.496920 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:32.497011 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:32.497272 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:32.497712 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:32.497815 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:32.497879 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 15:19:32.497928 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:19:32.503413 25892 raft_consensus.cc:1261] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Refusing update from remote peer dc163206f2f54bba9601ed069055e309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:32.503463 26026 raft_consensus.cc:1261] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Refusing update from remote peer dc163206f2f54bba9601ed069055e309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:32.504069 26095 consensus_queue.cc:1035] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:32.504259 26101 consensus_queue.cc:1035] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:32.517800 26105 mvcc.cc:205] Tried to move back new op lower bound from 7333979637767942144 to 7333979637699452928. Current Snapshot: MvccSnapshot[applied={T|T < 7333979637767942144 or (T in {7333979637767942144})}]
I20260927 15:19:32.519192 25607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:32.519340 25607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:32.521757 25607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Sending DeleteTablet for 3 replicas of tablet a82e76a9802741a5a526e55b96f0b209
I20260927 15:19:32.522605 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 25675
I20260927 15:19:32.522583 25872 tablet_service.cc:1514] Processing DeleteTablet for tablet a82e76a9802741a5a526e55b96f0b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:32 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:56282
I20260927 15:19:32.523211 26006 tablet_service.cc:1514] Processing DeleteTablet for tablet a82e76a9802741a5a526e55b96f0b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:32 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:33942
I20260927 15:19:32.523635 26115 tablet_replica.cc:330] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: stopping tablet replica
I20260927 15:19:32.523893 26115 raft_consensus.cc:2229] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:32.524149 26115 raft_consensus.cc:2258] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:32.525585 26115 ts_tablet_manager.cc:1905] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:32.528317 26115 ts_tablet_manager.cc:1918] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:32.528416 26115 log.cc:1196] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/a82e76a9802741a5a526e55b96f0b209
I20260927 15:19:32.528733 26115 ts_tablet_manager.cc:1939] T a82e76a9802741a5a526e55b96f0b209 P 72e1c6f5f12046d4867d24f854180773: Deleting consensus metadata
I20260927 15:19:32.528919 26114 tablet_replica.cc:330] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: stopping tablet replica
W20260927 15:19:32.529235 25597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:35629 (error 108)
I20260927 15:19:32.529275 26114 raft_consensus.cc:2229] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:32.529436 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 25809
I20260927 15:19:32.529594 26114 raft_consensus.cc:2258] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:32.530479 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.530175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35629 (local address 127.0.0.1:55016)
0927 15:19:32.530335 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.530393 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 15:19:32.530712 25597 catalog_manager.cc:4584] TS dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a82e76a9802741a5a526e55b96f0b209: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35629: connect: Connection refused (error 111)
I20260927 15:19:32.531361 25597 catalog_manager.cc:4841] TS 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599): tablet a82e76a9802741a5a526e55b96f0b209 (table test-table [id=640efb1f589245db957b040bf79eae16]) successfully deleted
I20260927 15:19:32.534487 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 25943
I20260927 15:19:32.535442 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:32.534714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:35841 (local address 127.0.0.1:56286)
0927 15:19:32.535310 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:32.535369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.130:35841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524}
I20260927 15:19:32.538946 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 25581
I20260927 15:19:52.544246 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36981
--webserver_interface=127.9.242.190
--webserver_port=35443
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:36981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:19:52.651762 26117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:52.652055 26117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:52.652098 26117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:52.656145 26117 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:19:52.656199 26117 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:19:52.656214 26117 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:19:52.656226 26117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:19:52.656242 26117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:52.656262 26117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:52.656293 26117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:19:52.656317 26117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:19:52.661548 26117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:36981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=35443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:52.662585 26117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:52.663607 26117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:52.676402 26126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:52.676455 26123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:52.676424 26117 server_base.cc:1034] running on GCE node
W20260927 15:19:52.676427 26124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:52.677325 26117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:52.678702 26117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:52.679903 26117 hybrid_clock.cc:648] HybridClock initialized: now 1790522392679891 us; error 26 us; skew 500 ppm
Sep 27 15:19:52 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:52.682953 26117 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:19:52.684069 26117 webserver.cc:460] Webserver started at http://127.9.242.190:35443/ using document root <none> and password file <none>
I20260927 15:19:52.684648 26117 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:52.684728 26117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:52.688668 26117 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:52.690915 26133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:52.692171 26117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:19:52.692282 26117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "a4f5d553952b46c4839c35abc15ee53f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "a4a8ae64d95bf2ce7133b8e48ba192ae"
server_key_iv: "85950a1bab3cad583a194aa73e394b8d"
server_key_version: "encryptionkey@0"
I20260927 15:19:52.692858 26117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:52.701759 26117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:52.704612 26117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:52.704772 26117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:52.711585 26117 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:36981
I20260927 15:19:52.711882 26185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:36981 every 8 connection(s)
I20260927 15:19:52.712770 26117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:19:52.717088 26186 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:19:52.718888 26186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Bootstrap starting.
I20260927 15:19:52.720659 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 26117
I20260927 15:19:52.721148 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:35629
--local_ip_for_outbound_sockets=127.9.242.129
--tserver_master_addrs=127.9.242.190:36981
--webserver_port=43733
--webserver_interface=127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:52.736371 26186 log.cc:824] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:52.745261 26186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:52.745620 26186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Bootstrap complete.
I20260927 15:19:52.749411 26186 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:52.749854 26186 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f5d553952b46c4839c35abc15ee53f, State: Initialized, Role: FOLLOWER
I20260927 15:19:52.750355 26186 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:52.750471 26186 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:19:52.750540 26186 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:19:52.750641 26186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 1 FOLLOWER]: Advancing to term 2
I20260927 15:19:52.752494 26186 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:52.752820 26186 leader_election.cc:302] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4f5d553952b46c4839c35abc15ee53f; no voters:
I20260927 15:19:52.753098 26186 leader_election.cc:288] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 15:19:52.753232 26190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 15:19:52.753475 26190 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 LEADER]: Becoming Leader. State: Replica: a4f5d553952b46c4839c35abc15ee53f, State: Running, Role: LEADER
I20260927 15:19:52.753880 26190 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:19:52.754290 26186 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:19:52.755236 26191 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a4f5d553952b46c4839c35abc15ee53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } } }
I20260927 15:19:52.755352 26191 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:52.755650 26192 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4f5d553952b46c4839c35abc15ee53f. Latest consensus state: current_term: 2 leader_uuid: "a4f5d553952b46c4839c35abc15ee53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } } }
I20260927 15:19:52.755796 26192 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: This master's current role is: LEADER
I20260927 15:19:52.756090 26199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:19:52.760775 26199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:19:52.761369 26199 catalog_manager.cc:1259] Loaded cluster ID: 17c733edf771470da9f8c28ba8eb17e5
I20260927 15:19:52.761427 26199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:19:52.763891 26199 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:19:52.764573 26199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Loaded TSK: 0
I20260927 15:19:52.778157 26199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Generated new TSK 1
I20260927 15:19:52.778435 26199 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:19:52.837503 26188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:52.837781 26188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:52.837829 26188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:52.841800 26188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:52.841856 26188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:52.841912 26188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:19:52.847083 26188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:35629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=43733
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:52.848053 26188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:52.849022 26188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:52.858706 26214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:52.858706 26216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:52.858778 26213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:52.859130 26188 server_base.cc:1034] running on GCE node
I20260927 15:19:52.859613 26188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:52.860471 26188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:52.861680 26188 hybrid_clock.cc:648] HybridClock initialized: now 1790522392861630 us; error 43 us; skew 500 ppm
Sep 27 15:19:52 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:52.865353 26188 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:19:52.866660 26188 webserver.cc:460] Webserver started at http://127.9.242.129:43733/ using document root <none> and password file <none>
I20260927 15:19:52.867413 26188 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:52.867528 26188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:52.873207 26188 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 15:19:52.876291 26223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:52.877877 26188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 15:19:52.878012 26188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "dc163206f2f54bba9601ed069055e309"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "3db71d0e34eba7cd53906579de2ed807"
server_key_iv: "a00a7a5381d0a31debe94efafcaae7a3"
server_key_version: "encryptionkey@0"
I20260927 15:19:52.878712 26188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:52.903887 26188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:52.907227 26188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:52.907406 26188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:52.907969 26188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:52.909032 26230 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:19:52.913244 26188 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:19:52.913321 26188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 15:19:52.913398 26188 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:19:52.915277 26188 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:19:52.915323 26188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:19:52.915629 26230 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Bootstrap starting.
I20260927 15:19:52.925109 26188 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:35629
I20260927 15:19:52.925953 26337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:35629 every 8 connection(s)
I20260927 15:19:52.926328 26188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:19:52.927907 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 26188
I20260927 15:19:52.928313 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:35841
--local_ip_for_outbound_sockets=127.9.242.130
--tserver_master_addrs=127.9.242.190:36981
--webserver_port=44143
--webserver_interface=127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:52.936156 26230 log.cc:824] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:19:52 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:52.944139 26230 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:52.944644 26230 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Bootstrap complete.
I20260927 15:19:52.945972 26230 ts_tablet_manager.cc:1397] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.014s
I20260927 15:19:52.946409 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:52.928667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:34521)
0927 15:19:52.929971 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:52.930042 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:52.931140 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:52.932466 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:52.932478 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:52.933022 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:52.933699 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:52.933721 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.935599 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.935602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:52.936796 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:52.936806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.937726 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.937730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:52.937940 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:52.938615 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:52.938636 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:52.940461 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:52.942940 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:52.942946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:52.942964 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:52.943293 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:52.943688 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:52.943691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:52.943693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:52.943858 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:52.944540 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:52.944547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:52.944835 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 15:19:52.945074 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 15:19:52.945394 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":141,"threads_started":1}
I20260927 15:19:52.946447 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:52.928580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:34521 (local address 127.9.242.190:36981)
0927 15:19:52.929249 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:52.929268 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:52.930192 (+ 924us) server_negotiation.cc:407] Connection header received
0927 15:19:52.931712 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:52.931724 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:52.932033 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:52.932309 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:52.933902 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.935428 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.936918 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.937545 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.937738 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:52.940679 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:52.940686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:52.940705 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:52.940742 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:52.942814 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:52.943424 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:52.943430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:52.943435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:52.943591 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:52.943966 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:52.943969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:52.943970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:52.944422 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:52.944548 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:52.945227 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 15:19:52.945531 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":109,"threads_started":1}
I20260927 15:19:52.948563 26230 raft_consensus.cc:348] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:52.949064 26230 raft_consensus.cc:729] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc163206f2f54bba9601ed069055e309, State: Initialized, Role: FOLLOWER
I20260927 15:19:52.949277 26338 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36981
I20260927 15:19:52.949517 26338 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:52.949548 26230 consensus_queue.cc:260] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:52.950076 26338 heartbeater.cc:511] Master 127.9.242.190:36981 requested a full tablet report, sending...
I20260927 15:19:52.950511 26230 ts_tablet_manager.cc:1428] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 15:19:52.952544 26150 ts_manager.cc:194] Registered new tserver with Master: dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629)
I20260927 15:19:52.956146 26150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:34521
I20260927 15:19:52.959050 26338 heartbeater.cc:503] Master 127.9.242.190:36981 was elected leader, sending a full tablet report...
Sep 27 15:19:52 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.129@KRBTEST.COM
I20260927 15:19:52.965301 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:52.955311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35629 (local address 127.0.0.1:54094)
0927 15:19:52.955818 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:52.955884 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:52.956010 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:52.956517 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:52.956522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:52.956956 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:52.957265 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:52.957274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.958325 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.958328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:52.958942 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:52.958951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.959676 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.959679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:52.959790 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:52.960293 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:52.960305 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:52.961775 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:52.963481 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:52.963490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:52.963496 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:52.963717 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:52.963981 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:52.963984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:52.963986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:52.964065 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:52.964641 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:52.964647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:52.964769 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 15:19:52.964886 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 15:19:52.965121 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260927 15:19:52.965301 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:52.955368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.9.242.129:35629)
0927 15:19:52.956057 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:52.956069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:52.956091 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:19:52.956157 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:52.956161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:52.956345 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:52.956491 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:52.957384 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.958220 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.959052 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:52.959495 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:52.959663 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:52.961919 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:52.961927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:52.961936 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:52.961957 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:52.963367 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:52.963820 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:52.963824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:52.963826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:52.963898 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:52.964142 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:52.964145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:52.964147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:52.964510 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:52.964658 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:52.964980 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 15:19:52.965136 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":597,"thread_start_us":96,"threads_started":1}
I20260927 15:19:52.966318 26256 tablet_service.cc:1514] Processing DeleteTablet for tablet a82e76a9802741a5a526e55b96f0b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:32 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:54094
I20260927 15:19:52.967101 26347 tablet_replica.cc:330] stopping tablet replica
I20260927 15:19:52.967396 26347 raft_consensus.cc:2229] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:52.967618 26347 raft_consensus.cc:2258] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:52.968539 26347 ts_tablet_manager.cc:1905] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:52.972020 26347 ts_tablet_manager.cc:1918] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:52.972100 26347 log.cc:1196] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal/wals/a82e76a9802741a5a526e55b96f0b209
I20260927 15:19:52.972333 26347 ts_tablet_manager.cc:1939] T a82e76a9802741a5a526e55b96f0b209 P dc163206f2f54bba9601ed069055e309: Deleting consensus metadata
I20260927 15:19:52.973939 26134 catalog_manager.cc:4841] TS dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629): tablet a82e76a9802741a5a526e55b96f0b209 (table test-table [id=640efb1f589245db957b040bf79eae16]) successfully deleted
W20260927 15:19:53.048236 26342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:53.048506 26342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:53.048563 26342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:53.052600 26342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:53.052667 26342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:53.052750 26342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:19:53.058329 26342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:35841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=44143
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:53.059331 26342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:53.060315 26342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:53.069496 26353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:53.069720 26342 server_base.cc:1034] running on GCE node
W20260927 15:19:53.069613 26355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:53.069499 26352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:53.070405 26342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:53.071103 26342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:53.072269 26342 hybrid_clock.cc:648] HybridClock initialized: now 1790522393072240 us; error 47 us; skew 500 ppm
Sep 27 15:19:53 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:53.074957 26342 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:19:53.075845 26342 webserver.cc:460] Webserver started at http://127.9.242.130:44143/ using document root <none> and password file <none>
I20260927 15:19:53.076361 26342 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:53.076439 26342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:53.080175 26342 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:53.082283 26362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:53.083587 26342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:53.083695 26342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "b6d2938b143c4a3da7a69abf30af0f15"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "a6f3f75e15aa839ec8faf45ffecb1e21"
server_key_iv: "e2b5b3f06b1849e6031679ae5f99fde6"
server_key_version: "encryptionkey@0"
I20260927 15:19:53.084350 26342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:53.098621 26342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:53.101578 26342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:53.101780 26342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:53.102349 26342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:53.103374 26369 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:19:53.107335 26342 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:19:53.107401 26342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 15:19:53.107465 26342 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:19:53.109078 26342 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:19:53.109122 26342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:19:53.109369 26369 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Bootstrap starting.
I20260927 15:19:53.118386 26342 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:35841
I20260927 15:19:53.118507 26476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:35841 every 8 connection(s)
I20260927 15:19:53.119359 26342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:19:53.126372 26369 log.cc:824] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:19:53 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:53.128301 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 26342
I20260927 15:19:53.128733 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:39599
--local_ip_for_outbound_sockets=127.9.242.131
--tserver_master_addrs=127.9.242.190:36981
--webserver_port=46589
--webserver_interface=127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:19:53.132035 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.121168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:51891 (local address 127.9.242.190:36981)
0927 15:19:53.121291 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.121295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.122046 (+ 751us) server_negotiation.cc:407] Connection header received
0927 15:19:53.122895 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.122898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.122960 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.123036 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:53.124191 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.124728 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.125869 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.126109 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.126198 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.128526 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:53.128529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:53.128531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:53.128564 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:53.129905 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.130489 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.130493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.130495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.130588 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.130929 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.130932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.130933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.131156 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:53.131249 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.131724 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.131863 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:19:53.132783 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.121360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:51891)
0927 15:19:53.121876 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.121920 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:53.122699 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:53.123197 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:53.123204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:53.123577 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:53.124045 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.124058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.124860 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.124862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.125761 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.125770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.126226 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.126229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.126391 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.126971 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:53.126986 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:53.128353 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:53.130010 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.130016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.130031 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.130363 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.130675 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.130678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.130680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.130840 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.131255 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:53.131261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:53.131499 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 15:19:53.131650 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 15:19:53.131896 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":149,"threads_started":1}
I20260927 15:19:53.133941 26369 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:19:53.134057 26477 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36981
I20260927 15:19:53.134290 26477 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:53.134294 26369 tablet_bootstrap.cc:492] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Bootstrap complete.
I20260927 15:19:53.134837 26477 heartbeater.cc:511] Master 127.9.242.190:36981 requested a full tablet report, sending...
I20260927 15:19:53.135232 26369 ts_tablet_manager.cc:1397] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Time spent bootstrapping tablet: real 0.026s user 0.025s sys 0.000s
I20260927 15:19:53.136767 26150 ts_manager.cc:194] Registered new tserver with Master: b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841)
I20260927 15:19:53.138056 26150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:51891
I20260927 15:19:53.137730 26369 raft_consensus.cc:348] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:53.138216 26369 raft_consensus.cc:729] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d2938b143c4a3da7a69abf30af0f15, State: Initialized, Role: FOLLOWER
I20260927 15:19:53.138707 26369 consensus_queue.cc:260] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } }
I20260927 15:19:53.139657 26369 ts_tablet_manager.cc:1428] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 15:19:53.140224 26477 heartbeater.cc:503] Master 127.9.242.190:36981 was elected leader, sending a full tablet report...
Sep 27 15:19:53 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522392, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.130@KRBTEST.COM
I20260927 15:19:53.147990 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.137729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:35841 (local address 127.0.0.1:34376)
0927 15:19:53.137831 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.137845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:53.137922 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:53.139169 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:53.139173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:53.139358 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:53.139473 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.139481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.141032 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.141035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.141848 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.141859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.142584 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.142587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.142657 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.143116 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:53.143126 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:53.144456 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:53.146508 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.146510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.146512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.146695 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.146968 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.146970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.146972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.147048 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.147555 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:53.147557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:53.147646 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 15:19:53.147739 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 15:19:53.147864 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 15:19:53.148298 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.137986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.9.242.130:35841)
0927 15:19:53.138534 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.138545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.138567 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:19:53.138630 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.138636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.138924 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.139625 (+ 701us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:53.139801 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.140919 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.141961 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.142479 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.142578 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.144620 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:53.144629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:53.144643 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:53.144668 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:53.146399 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.146792 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.146798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.146800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.146883 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.147120 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.147123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.147125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.147399 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:53.147538 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.147834 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.147974 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":108,"threads_started":1}
I20260927 15:19:53.149251 26411 tablet_service.cc:1514] Processing DeleteTablet for tablet a82e76a9802741a5a526e55b96f0b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:32 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:34376
I20260927 15:19:53.150094 26484 tablet_replica.cc:330] stopping tablet replica
I20260927 15:19:53.150406 26484 raft_consensus.cc:2229] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:53.150660 26484 raft_consensus.cc:2258] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:53.151687 26484 ts_tablet_manager.cc:1905] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:53.156216 26484 ts_tablet_manager.cc:1918] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:53.156306 26484 log.cc:1196] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/a82e76a9802741a5a526e55b96f0b209
I20260927 15:19:53.156549 26484 ts_tablet_manager.cc:1939] T a82e76a9802741a5a526e55b96f0b209 P b6d2938b143c4a3da7a69abf30af0f15: Deleting consensus metadata
I20260927 15:19:53.158288 26136 catalog_manager.cc:4841] TS b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841): tablet a82e76a9802741a5a526e55b96f0b209 (table test-table [id=640efb1f589245db957b040bf79eae16]) successfully deleted
W20260927 15:19:53.243774 26481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:19:53.244046 26481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:19:53.244091 26481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:19:53.248040 26481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:19:53.248096 26481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:19:53.248157 26481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:19:53.253324 26481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:39599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=46589
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:19:53.254370 26481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:19:53.255429 26481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:19:53.264688 26489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:19:53.264688 26492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:53.264868 26481 server_base.cc:1034] running on GCE node
W20260927 15:19:53.264694 26490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:19:53.265441 26481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:19:53.266228 26481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:19:53.267408 26481 hybrid_clock.cc:648] HybridClock initialized: now 1790522393267391 us; error 31 us; skew 500 ppm
Sep 27 15:19:53 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:19:53.269934 26481 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:19:53.270818 26481 webserver.cc:460] Webserver started at http://127.9.242.131:46589/ using document root <none> and password file <none>
I20260927 15:19:53.271356 26481 fs_manager.cc:359] Metadata directory not provided
I20260927 15:19:53.271433 26481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:19:53.275316 26481 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:19:53.277315 26499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:53.278592 26481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:19:53.278692 26481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "72e1c6f5f12046d4867d24f854180773"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "5285dbf988a1294a6260464b1a2b074f"
server_key_iv: "59aa938e76297ce9e97e4e01c5d6eda3"
server_key_version: "encryptionkey@0"
I20260927 15:19:53.279207 26481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:19:53.318746 26481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:19:53.321956 26481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:19:53.322137 26481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:19:53.322736 26481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:19:53.323570 26481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:19:53.323616 26481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:53.323664 26481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:19:53.323693 26481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:19:53.332389 26481 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:39599
I20260927 15:19:53.332422 26612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:39599 every 8 connection(s)
I20260927 15:19:53.333405 26481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:19:53.337204 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 26481
Sep 27 15:19:53 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:19:53.345979 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.337875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.9.242.190:36981)
0927 15:19:53.338074 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.338078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.338234 (+ 156us) server_negotiation.cc:407] Connection header received
0927 15:19:53.338324 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.338326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.338371 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.338472 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:53.339143 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.339758 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.340615 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.340829 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.340912 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.342572 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:53.342575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:53.342577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:53.342600 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:53.344149 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.344584 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.344588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.344590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.344702 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.345023 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.345026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.345028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.345235 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:53.345377 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.345649 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.345797 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260927 15:19:53.347949 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.335231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:60329 (local address 127.9.242.190:36981)
0927 15:19:53.335369 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.335373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.336261 (+ 888us) server_negotiation.cc:407] Connection header received
0927 15:19:53.337679 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.337683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.337741 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.337823 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:53.339377 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.339889 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.341083 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.341313 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.341406 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.344158 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:53.344161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:53.344163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:53.344186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:53.345874 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.346419 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.346422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.346424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.346506 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.346851 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.346854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.346855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.347060 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:53.347145 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.347715 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.347826 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:19:53.348788 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.335655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:60329)
0927 15:19:53.336114 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.336157 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:53.337412 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:53.338209 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:53.338219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:53.338695 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:19:53.339233 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.339251 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.340018 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.340021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.340972 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.340981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.341457 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.341462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.341694 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.342197 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:19:53.342219 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:19:53.343985 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:19:53.345983 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.345989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.346005 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.346331 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.346586 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:19:53.346589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:19:53.346591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:19:53.346762 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:19:53.347164 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:19:53.347171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:19:53.347418 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 15:19:53.347649 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 15:19:53.347980 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
I20260927 15:19:53.349906 26613 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36981
I20260927 15:19:53.350135 26613 heartbeater.cc:464] Registering TS with master...
I20260927 15:19:53.350683 26613 heartbeater.cc:511] Master 127.9.242.190:36981 requested a full tablet report, sending...
I20260927 15:19:53.351733 26150 ts_manager.cc:194] Registered new tserver with Master: 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599)
I20260927 15:19:53.352269 26150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:60329
I20260927 15:19:53.359586 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:19:53.363492 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.360242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.9.242.190:36981)
0927 15:19:53.360387 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.360392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.360531 (+ 139us) server_negotiation.cc:407] Connection header received
0927 15:19:53.360708 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.360710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.360811 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.360892 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:53.361079 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.361574 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.362409 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.362656 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.362741 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.363024 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:53.363160 (+ 136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 15:19:53.363373 (+ 213us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47254: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 15:19:53.363564 26340 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47254: token expired
W20260927 15:19:53.363983 26077 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.242.190:36981)
I20260927 15:19:53.370899 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.364292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.9.242.190:36981)
0927 15:19:53.364383 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.364386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.364396 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:19:53.364493 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.364496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.364539 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.364601 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:19:53.365228 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.365743 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.366556 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.366769 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.366850 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.368106 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:19:53.368109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:19:53.368111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:19:53.368134 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:19:53.369426 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.369929 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.369933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.369935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.370029 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:19:53.370246 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:19:53.370248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:19:53.370249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:19:53.370408 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:19:53.370523 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.370692 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.370783 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 15:19:53.373380 26150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:19:53.376715 26150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:19:53.384264 26411 tablet_service.cc:1467] Processing CreateTablet for tablet d4662ff096144a4e82d2354e4df3f9d4 (DEFAULT_TABLE table=test-table [id=6667f769900549c09ea41ad9985ea2d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:53.384269 26256 tablet_service.cc:1467] Processing CreateTablet for tablet d4662ff096144a4e82d2354e4df3f9d4 (DEFAULT_TABLE table=test-table [id=6667f769900549c09ea41ad9985ea2d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:53.384857 26256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4662ff096144a4e82d2354e4df3f9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:53.384857 26411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4662ff096144a4e82d2354e4df3f9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:53.387382 26230 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Bootstrap starting.
I20260927 15:19:53.387830 26369 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: Bootstrap starting.
I20260927 15:19:53.388268 26230 tablet_bootstrap.cc:654] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:53.388495 26369 tablet_bootstrap.cc:654] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:53.389262 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.383355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:39599 (local address 127.0.0.1:39706)
0927 15:19:53.383690 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.383704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:53.383775 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:53.384555 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:53.384558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:53.384577 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:53.384701 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.384708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.386242 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.386245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.387432 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.387443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.388803 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.388807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.388912 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.388971 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 15:19:53.389046 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 15:19:53.389129 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260927 15:19:53.389850 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.383547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.9.242.131:39599)
0927 15:19:53.384023 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.384035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.384058 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:19:53.384135 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.384140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.384359 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.384856 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:53.385024 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.386124 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.387741 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.388655 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.388762 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.388919 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.389577 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.389708 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":182,"threads_started":1}
I20260927 15:19:53.389891 26369 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: No bootstrap required, opened a new log
I20260927 15:19:53.389967 26369 ts_tablet_manager.cc:1397] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 15:19:53.390267 26230 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: No bootstrap required, opened a new log
I20260927 15:19:53.390347 26230 ts_tablet_manager.cc:1397] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 15:19:53.390401 26369 raft_consensus.cc:348] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.390532 26369 raft_consensus.cc:374] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:53.390574 26369 raft_consensus.cc:729] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d2938b143c4a3da7a69abf30af0f15, State: Initialized, Role: FOLLOWER
I20260927 15:19:53.390693 26369 consensus_queue.cc:260] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.390761 26230 raft_consensus.cc:348] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.390870 26230 raft_consensus.cc:374] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:53.390904 26230 raft_consensus.cc:729] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc163206f2f54bba9601ed069055e309, State: Initialized, Role: FOLLOWER
I20260927 15:19:53.390964 26369 ts_tablet_manager.cc:1428] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 15:19:53.391017 26230 consensus_queue.cc:260] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.391281 26230 ts_tablet_manager.cc:1428] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 15:19:53.392917 26547 tablet_service.cc:1467] Processing CreateTablet for tablet d4662ff096144a4e82d2354e4df3f9d4 (DEFAULT_TABLE table=test-table [id=6667f769900549c09ea41ad9985ea2d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:19:53.394079 26547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4662ff096144a4e82d2354e4df3f9d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:19:53.401072 26625 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Bootstrap starting.
I20260927 15:19:53.403537 26625 tablet_bootstrap.cc:654] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Neither blocks nor log segments found. Creating new log.
I20260927 15:19:53.404232 26625 log.cc:824] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Log is configured to *not* fsync() on all Append() calls
I20260927 15:19:53.406159 26625 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: No bootstrap required, opened a new log
I20260927 15:19:53.406361 26625 ts_tablet_manager.cc:1397] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:19:53.409184 26625 raft_consensus.cc:348] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.409423 26625 raft_consensus.cc:374] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:19:53.409457 26625 raft_consensus.cc:729] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e1c6f5f12046d4867d24f854180773, State: Initialized, Role: FOLLOWER
I20260927 15:19:53.409861 26625 consensus_queue.cc:260] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.410513 26613 heartbeater.cc:503] Master 127.9.242.190:36981 was elected leader, sending a full tablet report...
I20260927 15:19:53.410848 26625 ts_tablet_manager.cc:1428] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 15:19:53.427873 26339 tablet.cc:2369] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:53.547402 26483 raft_consensus.cc:482] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:19:53.547668 26483 raft_consensus.cc:504] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.548779 26483 leader_election.cc:288] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599), dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629)
I20260927 15:19:53.553488 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.549223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:46783 (local address 127.9.242.131:39599)
0927 15:19:53.549346 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.549351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.549365 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:19:53.549570 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.549574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.549625 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.549719 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:53.549988 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.550743 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.552287 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.552896 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.553025 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.553078 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.553271 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.553349 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 15:19:53.553522 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.549040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:39599 (local address 127.9.242.130:46783)
0927 15:19:53.549219 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.549242 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:53.549357 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:53.549748 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:53.549751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:53.549774 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:53.549892 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.549899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.550868 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.550870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.552133 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.552144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.553022 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.553025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.553106 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.553157 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 15:19:53.553243 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:19:53.553332 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 15:19:53.553696 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.549292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35629 (local address 127.9.242.130:36969)
0927 15:19:53.549723 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.549735 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:19:53.549827 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:19:53.550101 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:19:53.550103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:19:53.550119 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:19:53.550209 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.550213 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.551111 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.551115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.552269 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:19:53.552277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.553188 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.553192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:19:53.553288 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.553307 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:19:53.553363 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 15:19:53.553414 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"spinlock_wait_cycles":768,"thread_start_us":114,"threads_started":1}
I20260927 15:19:53.553562 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.549424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:36969 (local address 127.9.242.129:35629)
0927 15:19:53.549723 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.549728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.549792 (+ 64us) server_negotiation.cc:407] Connection header received
0927 15:19:53.549920 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.549923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.549984 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.550089 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:19:53.550301 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.551001 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.552385 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.553066 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.553172 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.553316 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.553376 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.553448 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
I20260927 15:19:53.554226 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4662ff096144a4e82d2354e4df3f9d4" candidate_uuid: "b6d2938b143c4a3da7a69abf30af0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e1c6f5f12046d4867d24f854180773" is_pre_election: true
I20260927 15:19:53.554286 26290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4662ff096144a4e82d2354e4df3f9d4" candidate_uuid: "b6d2938b143c4a3da7a69abf30af0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc163206f2f54bba9601ed069055e309" is_pre_election: true
I20260927 15:19:53.554579 26567 raft_consensus.cc:2454] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d2938b143c4a3da7a69abf30af0f15 in term 0.
I20260927 15:19:53.554615 26290 raft_consensus.cc:2454] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6d2938b143c4a3da7a69abf30af0f15 in term 0.
I20260927 15:19:53.555027 26363 leader_election.cc:302] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e1c6f5f12046d4867d24f854180773, b6d2938b143c4a3da7a69abf30af0f15; no voters:
I20260927 15:19:53.555259 26483 raft_consensus.cc:2781] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:19:53.555375 26483 raft_consensus.cc:482] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:19:53.555425 26483 raft_consensus.cc:3037] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:53.556551 26483 raft_consensus.cc:504] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.556862 26483 leader_election.cc:288] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [CANDIDATE]: Term 1 election: Requested vote from peers 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599), dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629)
I20260927 15:19:53.557235 26567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4662ff096144a4e82d2354e4df3f9d4" candidate_uuid: "b6d2938b143c4a3da7a69abf30af0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e1c6f5f12046d4867d24f854180773"
I20260927 15:19:53.557250 26290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4662ff096144a4e82d2354e4df3f9d4" candidate_uuid: "b6d2938b143c4a3da7a69abf30af0f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc163206f2f54bba9601ed069055e309"
I20260927 15:19:53.557368 26567 raft_consensus.cc:3037] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:53.557368 26290 raft_consensus.cc:3037] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:19:53.558583 26567 raft_consensus.cc:2454] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d2938b143c4a3da7a69abf30af0f15 in term 1.
I20260927 15:19:53.558583 26290 raft_consensus.cc:2454] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6d2938b143c4a3da7a69abf30af0f15 in term 1.
I20260927 15:19:53.558852 26363 leader_election.cc:302] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6d2938b143c4a3da7a69abf30af0f15, dc163206f2f54bba9601ed069055e309; no voters:
I20260927 15:19:53.559055 26483 raft_consensus.cc:2781] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:19:53.559249 26483 raft_consensus.cc:686] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 LEADER]: Becoming Leader. State: Replica: b6d2938b143c4a3da7a69abf30af0f15, State: Running, Role: LEADER
I20260927 15:19:53.559502 26483 consensus_queue.cc:237] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:19:53.562337 26149 catalog_manager.cc:5495] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "b6d2938b143c4a3da7a69abf30af0f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } health_report { overall_health: UNKNOWN } } }
W20260927 15:19:53.584403 26614 tablet.cc:2369] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:19:53.588675 26482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.585179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.9.242.130:35841)
0927 15:19:53.585314 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:19:53.585318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:19:53.585332 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:19:53.585424 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:19:53.585426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:19:53.585490 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:19:53.585583 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:19:53.585931 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.586447 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.587316 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:19:53.587569 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:19:53.587657 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:19:53.587870 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:19:53.588291 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:19:53.588373 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:19:53.588488 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 15:19:53.588538 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:19:53.592414 26567 raft_consensus.cc:1261] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Refusing update from remote peer b6d2938b143c4a3da7a69abf30af0f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:53.592432 26290 raft_consensus.cc:1261] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Refusing update from remote peer b6d2938b143c4a3da7a69abf30af0f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:19:53.593060 26630 consensus_queue.cc:1035] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:53.593267 26483 consensus_queue.cc:1035] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:19:53.599897 26633 mvcc.cc:205] Tried to move back new op lower bound from 7333979724148666368 to 7333979724021989376. Current Snapshot: MvccSnapshot[applied={T|T < 7333979724148666368 or (T in {7333979724148666368})}]
I20260927 15:19:53.603466 26635 mvcc.cc:205] Tried to move back new op lower bound from 7333979724148666368 to 7333979724021989376. Current Snapshot: MvccSnapshot[applied={T|T < 7333979724148666368}]
I20260927 15:19:53.606027 26148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:53.606160 26148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:19:53.608280 26148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Sending DeleteTablet for 3 replicas of tablet d4662ff096144a4e82d2354e4df3f9d4
I20260927 15:19:53.608870 26411 tablet_service.cc:1514] Processing DeleteTablet for tablet d4662ff096144a4e82d2354e4df3f9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:53 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:34376
I20260927 15:19:53.608974 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 26188
I20260927 15:19:53.609056 26484 tablet_replica.cc:330] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: stopping tablet replica
I20260927 15:19:53.609165 26484 raft_consensus.cc:2229] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:19:53.609126 26547 tablet_service.cc:1514] Processing DeleteTablet for tablet d4662ff096144a4e82d2354e4df3f9d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:19:53 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:39706
I20260927 15:19:53.609422 26484 raft_consensus.cc:2258] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:53.609947 26642 tablet_replica.cc:330] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: stopping tablet replica
I20260927 15:19:53.610224 26642 raft_consensus.cc:2229] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:19:53.610396 26484 ts_tablet_manager.cc:1905] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:53.610513 26642 raft_consensus.cc:2258] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:19:53.612866 26484 ts_tablet_manager.cc:1918] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:53.612931 26484 log.cc:1196] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/d4662ff096144a4e82d2354e4df3f9d4
I20260927 15:19:53.613152 26484 ts_tablet_manager.cc:1939] T d4662ff096144a4e82d2354e4df3f9d4 P b6d2938b143c4a3da7a69abf30af0f15: Deleting consensus metadata
I20260927 15:19:53.613852 26136 catalog_manager.cc:4841] TS b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841): tablet d4662ff096144a4e82d2354e4df3f9d4 (table test-table [id=6667f769900549c09ea41ad9985ea2d3]) successfully deleted
W20260927 15:19:53.615244 26134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:35629 (error 108)
I20260927 15:19:53.615345 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 26342
I20260927 15:19:53.616621 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:19:53.616338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35629 (local address 127.0.0.1:54102)
0927 15:19:53.616464 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 15:19:53.616539 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 15:19:53.616940 26134 catalog_manager.cc:4584] TS dc163206f2f54bba9601ed069055e309 (127.9.242.129:35629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4662ff096144a4e82d2354e4df3f9d4: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35629: connect: Connection refused (error 111)
I20260927 15:19:53.617487 26642 ts_tablet_manager.cc:1905] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:19:53.620085 26642 ts_tablet_manager.cc:1918] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:19:53.620179 26642 log.cc:1196] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/d4662ff096144a4e82d2354e4df3f9d4
I20260927 15:19:53.620499 26642 ts_tablet_manager.cc:1939] T d4662ff096144a4e82d2354e4df3f9d4 P 72e1c6f5f12046d4867d24f854180773: Deleting consensus metadata
I20260927 15:19:53.621763 26134 catalog_manager.cc:4841] TS 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599): tablet d4662ff096144a4e82d2354e4df3f9d4 (table test-table [id=6667f769900549c09ea41ad9985ea2d3]) successfully deleted
I20260927 15:19:53.622057 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 26481
I20260927 15:19:53.626667 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 26117
I20260927 15:20:13.631947 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36981
--webserver_interface=127.9.242.190
--webserver_port=35443
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:36981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:13.741484 26644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:13.741801 26644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:13.741871 26644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:13.746042 26644 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 15:20:13.746114 26644 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 15:20:13.746134 26644 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 15:20:13.746153 26644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:13.746176 26644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:13.746202 26644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:13.746233 26644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:13.746258 26644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:13.751663 26644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:36981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=35443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:13.752735 26644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:13.753787 26644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:20:13.762241 26644 server_base.cc:1034] running on GCE node
W20260927 15:20:13.762204 26652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.762204 26649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.762208 26650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:13.763067 26644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:13.764143 26644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:13.765324 26644 hybrid_clock.cc:648] HybridClock initialized: now 1790522413765291 us; error 45 us; skew 500 ppm
Sep 27 15:20:13 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:13.768085 26644 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:20:13.768922 26644 webserver.cc:460] Webserver started at http://127.9.242.190:35443/ using document root <none> and password file <none>
I20260927 15:20:13.769444 26644 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:13.769511 26644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:13.773386 26644 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:13.775585 26659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:13.776767 26644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:13.776880 26644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "a4f5d553952b46c4839c35abc15ee53f"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "a4a8ae64d95bf2ce7133b8e48ba192ae"
server_key_iv: "85950a1bab3cad583a194aa73e394b8d"
server_key_version: "encryptionkey@0"
I20260927 15:20:13.777449 26644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:13.803161 26644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:13.806167 26644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:13.806317 26644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:13.813172 26644 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:36981
I20260927 15:20:13.813442 26711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:36981 every 8 connection(s)
I20260927 15:20:13.814332 26644 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:13.818177 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 26644
I20260927 15:20:13.818593 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:35629
--local_ip_for_outbound_sockets=127.9.242.129
--tserver_master_addrs=127.9.242.190:36981
--webserver_port=43733
--webserver_interface=127.9.242.129
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:13.818634 26712 sys_catalog.cc:263] Verifying existing consensus state
I20260927 15:20:13.820269 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Bootstrap starting.
I20260927 15:20:13.843295 26712 log.cc:824] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:13.857849 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:20:13.858301 26712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Bootstrap complete.
I20260927 15:20:13.863929 26712 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:20:13.864454 26712 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4f5d553952b46c4839c35abc15ee53f, State: Initialized, Role: FOLLOWER
I20260927 15:20:13.865016 26712 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:20:13.865156 26712 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:13.865226 26712 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:13.865334 26712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 2 FOLLOWER]: Advancing to term 3
I20260927 15:20:13.867982 26712 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:20:13.868345 26712 leader_election.cc:302] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4f5d553952b46c4839c35abc15ee53f; no voters:
I20260927 15:20:13.868701 26712 leader_election.cc:288] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 15:20:13.868826 26716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 3 FOLLOWER]: Leader election won for term 3
I20260927 15:20:13.869088 26716 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [term 3 LEADER]: Becoming Leader. State: Replica: a4f5d553952b46c4839c35abc15ee53f, State: Running, Role: LEADER
I20260927 15:20:13.869443 26716 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } }
I20260927 15:20:13.869961 26712 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:13.870891 26718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4f5d553952b46c4839c35abc15ee53f. Latest consensus state: current_term: 3 leader_uuid: "a4f5d553952b46c4839c35abc15ee53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } } }
I20260927 15:20:13.870900 26717 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a4f5d553952b46c4839c35abc15ee53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4f5d553952b46c4839c35abc15ee53f" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36981 } } }
I20260927 15:20:13.871028 26718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:13.871028 26717 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:13.871435 26725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:13.876367 26725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:13.877107 26725 catalog_manager.cc:1259] Loaded cluster ID: 17c733edf771470da9f8c28ba8eb17e5
I20260927 15:20:13.877159 26725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:13.878937 26725 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:13.879523 26725 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Loaded TSK: 0
I20260927 15:20:13.879588 26725 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a4f5d553952b46c4839c35abc15ee53f: Loaded TSK: 1
F20260927 15:20:13.880201 26725 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2ae536e9ad google::LogMessage::Fail() at ??:0
@ 0x7f2ae537188a google::LogMessage::SendToLog() at ??:0
@ 0x7f2ae536e4de google::LogMessage::Flush() at ??:0
@ 0x7f2ae536fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2ae803218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2ae802af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2aefcd37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2aefcad99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2aefcae178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f2aefcefdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2aefd0e82b std::function<>::operator()() at ??:0
@ 0x7f2aefcadb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2aefcaef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2aefcab579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2aefcef640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2ae75c03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2ae75c0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2ae75c23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2ae75b1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2ae8bf56db start_thread at ??:0
@ 0x7f2ae3e6c61f clone at ??:0
W20260927 15:20:13.977017 26714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:13.977263 26714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:13.977306 26714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:13.981343 26714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:13.981403 26714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:13.981465 26714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:20:13.986879 26714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:35629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=43733
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:13.988020 26714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:13.989034 26714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:13.998582 26760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.998592 26759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:13.998585 26762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:13.999056 26714 server_base.cc:1034] running on GCE node
I20260927 15:20:13.999578 26714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:14.000481 26714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:14.001709 26714 hybrid_clock.cc:648] HybridClock initialized: now 1790522414001673 us; error 47 us; skew 500 ppm
Sep 27 15:20:13 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:14.005307 26714 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:20:14.006606 26714 webserver.cc:460] Webserver started at http://127.9.242.129:43733/ using document root <none> and password file <none>
I20260927 15:20:14.007372 26714 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:14.007471 26714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:14.012868 26714 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:20:14.015748 26769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.017457 26714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:14.017596 26714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "dc163206f2f54bba9601ed069055e309"
format_stamp: "Formatted at 2026-09-27 15:19:31 on dist-test-slave-m4lt"
server_key: "3db71d0e34eba7cd53906579de2ed807"
server_key_iv: "a00a7a5381d0a31debe94efafcaae7a3"
server_key_version: "encryptionkey@0"
I20260927 15:20:14.018421 26714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:14.040539 26714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:14.044394 26714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:14.044605 26714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:14.045365 26714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:14.046761 26776 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 15:20:14.050762 26714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 15:20:14.050841 26714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 15:20:14.050906 26714 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 15:20:14.053119 26714 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 15:20:14.053182 26714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 15:20:14.053562 26776 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Bootstrap starting.
I20260927 15:20:14.063499 26714 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:35629
I20260927 15:20:14.063686 26883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:35629 every 8 connection(s)
I20260927 15:20:14.064494 26714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:14.065933 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 26714
I20260927 15:20:14.066326 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:35841
--local_ip_for_outbound_sockets=127.9.242.130
--tserver_master_addrs=127.9.242.190:36981
--webserver_port=44143
--webserver_interface=127.9.242.130
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:14.073849 26776 log.cc:824] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:14.080595 26776 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 15:20:14.080943 26776 tablet_bootstrap.cc:492] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Bootstrap complete.
I20260927 15:20:14.081890 26776 ts_tablet_manager.cc:1397] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.010s
I20260927 15:20:14.084250 26776 raft_consensus.cc:348] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:20:14.084656 26776 raft_consensus.cc:729] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc163206f2f54bba9601ed069055e309, State: Initialized, Role: FOLLOWER
I20260927 15:20:14.085119 26776 consensus_queue.cc:260] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:20:14.086047 26776 ts_tablet_manager.cc:1428] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260927 15:20:14.105629 26886 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:20:14.066644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:34561)
0927 15:20:14.067875 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.067939 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:14.068977 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:14.104720 (+ 35743us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":147,"threads_started":1}
W20260927 15:20:14.106197 26884 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:36981 (0 consecutive failures): Network error: Failed to ping master at 127.9.242.190:36981: Client connection negotiation failed: client connection to 127.9.242.190:36981: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 15:20:14.106871 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.106636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:45271)
0927 15:20:14.106757 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.106821 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 15:20:14.107427 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.107238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:56235)
0927 15:20:14.107337 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.107380 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 15:20:14.107574 26884 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:20:14.186815 26887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:14.187089 26887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:14.187150 26887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:14.191123 26887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:14.191183 26887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:14.191262 26887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:20:14.196388 26887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:35841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=44143
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:14.197402 26887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:14.198390 26887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:14.207494 26894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:14.207491 26895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:14.207561 26897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:14.207715 26887 server_base.cc:1034] running on GCE node
I20260927 15:20:14.208305 26887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:14.208937 26887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:14.210119 26887 hybrid_clock.cc:648] HybridClock initialized: now 1790522414210104 us; error 24 us; skew 500 ppm
Sep 27 15:20:14 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522414, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:14.212694 26887 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:20:14.213608 26887 webserver.cc:460] Webserver started at http://127.9.242.130:44143/ using document root <none> and password file <none>
I20260927 15:20:14.214159 26887 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:14.214219 26887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:14.217916 26887 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 15:20:14.219961 26904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.221302 26887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:14.221410 26887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "b6d2938b143c4a3da7a69abf30af0f15"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "a6f3f75e15aa839ec8faf45ffecb1e21"
server_key_iv: "e2b5b3f06b1849e6031679ae5f99fde6"
server_key_version: "encryptionkey@0"
I20260927 15:20:14.222067 26887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:14.254959 26887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:14.258090 26887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:14.258266 26887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:14.258853 26887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:14.259650 26887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:14.259697 26887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.259750 26887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:14.259780 26887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.268363 26887 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:35841
I20260927 15:20:14.268394 27017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:35841 every 8 connection(s)
I20260927 15:20:14.269402 26887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:14.273355 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.271681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:34071)
0927 15:20:14.272179 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.272623 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":127,"threads_started":1}
W20260927 15:20:14.273797 27018 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:36981 (0 consecutive failures): Network error: Failed to ping master at 127.9.242.190:36981: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
I20260927 15:20:14.274317 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 26887
I20260927 15:20:14.274451 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.274133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:36837)
0927 15:20:14.274252 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.274306 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:20:14.274772 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:39599
--local_ip_for_outbound_sockets=127.9.242.131
--tserver_master_addrs=127.9.242.190:36981
--webserver_port=46589
--webserver_interface=127.9.242.131
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:20:14.275025 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.274806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:58529)
0927 15:20:14.274917 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.274963 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:20:14.275173 27018 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 15:20:14.383582 27021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:14.383860 27021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:14.383905 27021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:14.387982 27021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:14.388054 27021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:14.388113 27021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:20:14.393540 27021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:39599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=46589
--tserver_master_addrs=127.9.242.190:36981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:14.394622 27021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:14.395583 27021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:14.404940 27027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:14.404942 27029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:14.404997 27026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:14.405519 27021 server_base.cc:1034] running on GCE node
I20260927 15:20:14.405947 27021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:14.406628 27021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:14.407795 27021 hybrid_clock.cc:648] HybridClock initialized: now 1790522414407766 us; error 40 us; skew 500 ppm
Sep 27 15:20:14 dist-test-slave-m4lt krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522414, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:14.410495 27021 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:20:14.411501 27021 webserver.cc:460] Webserver started at http://127.9.242.131:46589/ using document root <none> and password file <none>
I20260927 15:20:14.412092 27021 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:14.412163 27021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:14.416272 27021 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:14.418421 27036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.419675 27021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:14.419801 27021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "72e1c6f5f12046d4867d24f854180773"
format_stamp: "Formatted at 2026-09-27 15:19:32 on dist-test-slave-m4lt"
server_key: "5285dbf988a1294a6260464b1a2b074f"
server_key_iv: "59aa938e76297ce9e97e4e01c5d6eda3"
server_key_version: "encryptionkey@0"
I20260927 15:20:14.420351 27021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:14.453011 27021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:14.455892 27021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:14.456079 27021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:14.456658 27021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:14.457468 27021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:14.457517 27021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.457571 27021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:14.457602 27021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:14.466265 27021 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:39599
I20260927 15:20:14.466333 27149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:39599 every 8 connection(s)
I20260927 15:20:14.467259 27021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:14.471024 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.469305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:44537)
0927 15:20:14.469790 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.470215 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260927 15:20:14.471174 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27021
W20260927 15:20:14.471488 27150 heartbeater.cc:644] Failed to heartbeat to 127.9.242.190:36981 (0 consecutive failures): Network error: Failed to ping master at 127.9.242.190:36981: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
I20260927 15:20:14.472070 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.471832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:45923)
0927 15:20:14.471960 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.472017 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 15:20:14.472615 27152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:14.472431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:37403)
0927 15:20:14.472529 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:14.472569 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:20:14.472764 27150 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 15:20:15.108539 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.108088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:46779)
0927 15:20:15.108330 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.108397 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260927 15:20:15.276095 27155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.275697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:58709)
0927 15:20:15.275941 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.275997 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260927 15:20:15.473728 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.473289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:50811)
0927 15:20:15.473522 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.473575 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260927 15:20:15.862883 27157 raft_consensus.cc:482] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:15.863144 27157 raft_consensus.cc:504] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:20:15.864236 27157 leader_election.cc:288] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599), b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841)
Sep 27 15:20:15 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.131@KRBTEST.COM
Sep 27 15:20:15 dist-test-slave-m4lt krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.130@KRBTEST.COM
I20260927 15:20:15.876803 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.864438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:39599 (local address 127.9.242.129:54299)
0927 15:20:15.864740 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.864754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:15.864870 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:15.866423 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:15.866428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:15.866938 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:15.867385 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.867401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.868607 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.868610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.869525 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.869534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.870226 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.870230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.870441 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.871056 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:15.871078 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:15.872654 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:15.874567 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.874573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.874586 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.874820 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.875139 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.875142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.875144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.875270 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.875879 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:15.875886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:15.876096 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 15:20:15.876257 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:20:15.876582 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":109,"spinlock_wait_cycles":29440,"thread_start_us":83,"threads_started":1}
I20260927 15:20:15.876811 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.864596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:54299 (local address 127.9.242.131:39599)
0927 15:20:15.864970 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:15.864995 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:15.865032 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:20:15.865754 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:15.865760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:15.866076 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:15.866344 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:15.867544 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.868482 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.869636 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.870115 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.870264 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.872850 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:15.872856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:15.872867 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:15.872894 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:15.874448 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.874924 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.874929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.874933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.875044 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.875353 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.875357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.875358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.875761 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:15.875870 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:15.876379 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 15:20:15.876632 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":69,"threads_started":1}
I20260927 15:20:15.876811 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.864725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:37707 (local address 127.9.242.130:35841)
0927 15:20:15.865058 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:15.865068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:15.865097 (+ 29us) server_negotiation.cc:407] Connection header received
0927 15:20:15.865725 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:15.865732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:15.866075 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:15.866345 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:15.867544 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.868473 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.869633 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.870119 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.870262 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.873104 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:15.873108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:15.873120 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:15.873146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:15.874702 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.875081 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.875087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.875092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.875195 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:15.875433 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:15.875437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:15.875438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:15.875797 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:15.875902 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:15.876380 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 15:20:15.876627 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":57,"threads_started":1}
I20260927 15:20:15.876832 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:15.864496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:35841 (local address 127.9.242.129:37707)
0927 15:20:15.864802 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:15.864815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:15.864909 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:15.866422 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:15.866428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:15.866939 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:15.867385 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.867400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.868613 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.868615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.869517 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:15.869526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:15.870231 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:15.870235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:15.870440 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:15.871333 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:15.871347 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:15.872935 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:15.874805 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.874808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.874810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.874986 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.875286 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:15.875289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:15.875291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:15.875359 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:15.875904 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:15.875907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:15.876096 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 15:20:15.876257 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:20:15.876657 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":246,"spinlock_wait_cycles":384,"thread_start_us":56,"threads_started":1}
W20260927 15:20:15.879323 26770 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
W20260927 15:20:15.879527 26772 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
I20260927 15:20:15.879601 26772 leader_election.cc:302] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc163206f2f54bba9601ed069055e309; no voters: 72e1c6f5f12046d4867d24f854180773, b6d2938b143c4a3da7a69abf30af0f15
I20260927 15:20:15.879846 27157 raft_consensus.cc:2726] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:20:16.109625 27158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.109325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:53771)
0927 15:20:16.109477 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.109540 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 15:20:16.277172 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.276750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:43047)
0927 15:20:16.277013 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.277069 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
I20260927 15:20:16.474807 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:16.474387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:47511)
0927 15:20:16.474661 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:16.474719 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
I20260927 15:20:17.110836 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:17.110414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:43623)
0927 15:20:17.110676 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:17.110738 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260927 15:20:17.278236 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:17.277863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:44571)
0927 15:20:17.278089 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:17.278141 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260927 15:20:17.475848 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:17.475436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:44639)
0927 15:20:17.475689 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:17.475743 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260927 15:20:18.080076 27167 raft_consensus.cc:482] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:18.080226 27167 raft_consensus.cc:504] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:20:18.080696 27167 leader_election.cc:288] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599), b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841)
W20260927 15:20:18.081487 26770 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
W20260927 15:20:18.081625 26772 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
I20260927 15:20:18.081730 26772 leader_election.cc:302] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc163206f2f54bba9601ed069055e309; no voters: 72e1c6f5f12046d4867d24f854180773, b6d2938b143c4a3da7a69abf30af0f15
I20260927 15:20:18.081856 27167 raft_consensus.cc:2726] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:20:18.112032 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.111615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:44691)
0927 15:20:18.111869 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.111927 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260927 15:20:18.279350 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.278914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:56273)
0927 15:20:18.279177 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.279244 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260927 15:20:18.477047 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:18.476623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:40069)
0927 15:20:18.476863 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:18.476955 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260927 15:20:19.113389 27171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:19.112960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:43431)
0927 15:20:19.113234 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:19.113291 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":70,"threads_started":1}
I20260927 15:20:19.280540 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:19.280102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:36125)
0927 15:20:19.280376 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:19.280443 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260927 15:20:19.478209 27173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:19.477817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:46707)
0927 15:20:19.478059 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:19.478117 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260927 15:20:20.114600 27174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.114182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:33693)
0927 15:20:20.114440 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.114503 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260927 15:20:20.281862 27175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.281388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:58113)
0927 15:20:20.281700 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.281761 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260927 15:20:20.479470 27176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:20.479013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:53839)
0927 15:20:20.479308 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:20.479372 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260927 15:20:21.059900 27177 raft_consensus.cc:482] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:21.060058 27177 raft_consensus.cc:504] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:20:21.060632 27177 leader_election.cc:288] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599), b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841)
W20260927 15:20:21.061262 26772 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
W20260927 15:20:21.061380 26770 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
I20260927 15:20:21.061452 26770 leader_election.cc:302] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc163206f2f54bba9601ed069055e309; no voters: 72e1c6f5f12046d4867d24f854180773, b6d2938b143c4a3da7a69abf30af0f15
I20260927 15:20:21.061594 27177 raft_consensus.cc:2726] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:20:21.115816 27178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:21.115382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:51401)
0927 15:20:21.115657 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:21.115719 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":97,"threads_started":1}
I20260927 15:20:21.283061 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:21.282604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:47899)
0927 15:20:21.282894 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:21.282964 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
I20260927 15:20:21.480634 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:21.480197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:43477)
0927 15:20:21.480482 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:21.480538 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
I20260927 15:20:22.117172 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:22.116698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:42717)
0927 15:20:22.117003 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:22.117069 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
I20260927 15:20:22.284318 27182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:22.283847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:50839)
0927 15:20:22.284153 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:22.284218 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260927 15:20:22.481961 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:22.481482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:40963)
0927 15:20:22.481793 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:22.481860 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":72,"threads_started":1}
I20260927 15:20:23.118300 27184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:23.117917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:40003)
0927 15:20:23.118149 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:23.118204 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260927 15:20:23.285593 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:23.285100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:40319)
0927 15:20:23.285424 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:23.285495 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":82,"threads_started":1}
I20260927 15:20:23.429265 27186 raft_consensus.cc:482] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:23.429404 27186 raft_consensus.cc:504] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72e1c6f5f12046d4867d24f854180773" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 39599 } } peers { permanent_uuid: "b6d2938b143c4a3da7a69abf30af0f15" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 35841 } } peers { permanent_uuid: "dc163206f2f54bba9601ed069055e309" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35629 } }
I20260927 15:20:23.429858 27186 leader_election.cc:288] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599), b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841)
W20260927 15:20:23.430581 26772 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b6d2938b143c4a3da7a69abf30af0f15 (127.9.242.130:35841): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
W20260927 15:20:23.430683 26770 leader_election.cc:341] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 72e1c6f5f12046d4867d24f854180773 (127.9.242.131:39599): Not found: Tablet not found: d4662ff096144a4e82d2354e4df3f9d4
I20260927 15:20:23.430770 26770 leader_election.cc:302] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc163206f2f54bba9601ed069055e309; no voters: 72e1c6f5f12046d4867d24f854180773, b6d2938b143c4a3da7a69abf30af0f15
I20260927 15:20:23.430922 27186 raft_consensus.cc:2726] T d4662ff096144a4e82d2354e4df3f9d4 P dc163206f2f54bba9601ed069055e309 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 15:20:23.483099 27187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:23.482668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:56629)
0927 15:20:23.482941 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:23.482998 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260927 15:20:24.119589 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.119170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:41283)
0927 15:20:24.119428 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:24.119491 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260927 15:20:24.286782 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.286393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:45849)
0927 15:20:24.286638 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:24.286689 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260927 15:20:24.484252 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:24.483833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:49073)
0927 15:20:24.484098 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:24.484155 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":70,"threads_started":1}
I20260927 15:20:25.120791 27191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:25.120349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:50517)
0927 15:20:25.120610 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:25.120667 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260927 15:20:25.287926 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:25.287479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:40069)
0927 15:20:25.287762 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:25.287829 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260927 15:20:25.485523 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:25.485117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:60411)
0927 15:20:25.485369 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:25.485429 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260927 15:20:26.122025 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.121579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:32797)
0927 15:20:26.121857 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.121924 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
I20260927 15:20:26.289098 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.288696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:56879)
0927 15:20:26.288947 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.289003 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260927 15:20:26.486706 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:26.486206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:37117)
0927 15:20:26.486489 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:26.486559 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260927 15:20:27.123353 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.122848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:55145)
0927 15:20:27.123139 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.123231 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260927 15:20:27.290351 27198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.289910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:46479)
0927 15:20:27.290185 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.290250 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260927 15:20:27.487883 27199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:27.487466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:58631)
0927 15:20:27.487717 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:27.487781 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260927 15:20:28.124640 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:28.124195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:49199)
0927 15:20:28.124465 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:28.124527 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260927 15:20:28.291556 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:28.291133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:52675)
0927 15:20:28.291397 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:28.291461 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260927 15:20:28.489017 27202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:28.488609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.131:41533)
0927 15:20:28.488863 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:28.488920 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260927 15:20:29.126135 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:29.125647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.129:35561)
0927 15:20:29.125955 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:29.126027 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260927 15:20:29.292807 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:29.292394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36981 (local address 127.9.242.130:55595)
0927 15:20:29.292651 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:29.292704 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.190:36981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260927 15:20:29.475549 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 26714
I20260927 15:20:29.480445 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 26887
I20260927 15:20:29.484815 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27021
2026-09-27T15:20:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59033 ms)
[----------] 30 tests from SecurityITest (235385 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:29 dist-test-slave-m4lt krb5kdc[27207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:29 dist-test-slave-m4lt krb5kdc[27207](info): set up 2 sockets
Sep 27 15:20:29 dist-test-slave-m4lt krb5kdc[27207](info): commencing operation
krb5kdc: starting...
W20260927 15:20:30.449333 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:30 dist-test-slave-m4lt krb5kdc[27207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:20:30.565969 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34565
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:34565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:30.672883 27220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:30.673146 27220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:30.673203 27220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:30.677201 27220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:30.677263 27220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:30.677289 27220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:30.677307 27220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:30.677328 27220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:30.682467 27220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:34565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:30.683466 27220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:30.684432 27220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:30.692658 27228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:30.692740 27220 server_base.cc:1034] running on GCE node
W20260927 15:20:30.692698 27226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:30.692673 27225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:30.693449 27220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:30.694535 27220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:30.695688 27220 hybrid_clock.cc:648] HybridClock initialized: now 1790522430695681 us; error 46 us; skew 500 ppm
Sep 27 15:20:30 dist-test-slave-m4lt krb5kdc[27207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:30.698401 27220 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:20:30.699277 27220 webserver.cc:460] Webserver started at http://127.9.242.190:45201/ using document root <none> and password file <none>
I20260927 15:20:30.699820 27220 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:30.699880 27220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:30.700037 27220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:30.702062 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "76f6eacf6aa046f281812856f97091a4"
format_stamp: "Formatted at 2026-09-27 15:20:30 on dist-test-slave-m4lt"
server_key: "f5cb18322678b894c5251c195173cd23"
server_key_iv: "b3c6b0239f5c37776f02c63f9c6f7886"
server_key_version: "encryptionkey@0"
I20260927 15:20:30.702553 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "76f6eacf6aa046f281812856f97091a4"
format_stamp: "Formatted at 2026-09-27 15:20:30 on dist-test-slave-m4lt"
server_key: "f5cb18322678b894c5251c195173cd23"
server_key_iv: "b3c6b0239f5c37776f02c63f9c6f7886"
server_key_version: "encryptionkey@0"
I20260927 15:20:30.706097 27220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:30.708484 27235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:30.709693 27220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:30.709836 27220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "76f6eacf6aa046f281812856f97091a4"
format_stamp: "Formatted at 2026-09-27 15:20:30 on dist-test-slave-m4lt"
server_key: "f5cb18322678b894c5251c195173cd23"
server_key_iv: "b3c6b0239f5c37776f02c63f9c6f7886"
server_key_version: "encryptionkey@0"
I20260927 15:20:30.709950 27220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:30.724514 27220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:30.727265 27220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:30.727442 27220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:30.734236 27220 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:34565
I20260927 15:20:30.734252 27287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:34565 every 8 connection(s)
I20260927 15:20:30.735211 27220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:30.738082 27288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:30.741871 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27220
I20260927 15:20:30.741983 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:30.742280 10186 external_mini_cluster.cc:1428] Setting key dfe132180c5292beef0f36337b59e709
I20260927 15:20:30.743974 27288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4: Bootstrap starting.
I20260927 15:20:30.746479 27288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:30.747210 27288 log.cc:824] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:30.749013 27288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4: No bootstrap required, opened a new log
Sep 27 15:20:30 dist-test-slave-m4lt krb5kdc[27207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:30.752697 27288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6eacf6aa046f281812856f97091a4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34565 } }
I20260927 15:20:30.753033 27288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:30.753108 27288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f6eacf6aa046f281812856f97091a4, State: Initialized, Role: FOLLOWER
I20260927 15:20:30.753695 27288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6eacf6aa046f281812856f97091a4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34565 } }
I20260927 15:20:30.753854 27288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:30.753932 27288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:30.754053 27288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:30.755162 27288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6eacf6aa046f281812856f97091a4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34565 } }
I20260927 15:20:30.755522 27288 leader_election.cc:302] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76f6eacf6aa046f281812856f97091a4; no voters:
I20260927 15:20:30.755880 27288 leader_election.cc:288] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:30.756089 27293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:30.756410 27293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [term 1 LEADER]: Becoming Leader. State: Replica: 76f6eacf6aa046f281812856f97091a4, State: Running, Role: LEADER
I20260927 15:20:30.756744 27293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6eacf6aa046f281812856f97091a4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34565 } }
I20260927 15:20:30.757120 27288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:30.757174 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:30.743728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.9.242.190:34565)
0927 15:20:30.744143 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:30.744159 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:30.744195 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:20:30.744820 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:30.744869 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:30.745208 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:30.745492 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:30.746430 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:30.747505 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:30.748355 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:30.748855 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:30.748993 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:30.751521 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:30.751526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:30.751540 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:30.751566 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:30.753326 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:30.753875 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:30.753881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:30.753887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:30.754022 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:30.754367 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:30.754370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:30.754371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:30.754924 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:30.755117 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:30.755786 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 15:20:30.756183 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260927 15:20:30.758400 27295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76f6eacf6aa046f281812856f97091a4. Latest consensus state: current_term: 1 leader_uuid: "76f6eacf6aa046f281812856f97091a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6eacf6aa046f281812856f97091a4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34565 } } }
I20260927 15:20:30.758496 27295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:30.758756 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76f6eacf6aa046f281812856f97091a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76f6eacf6aa046f281812856f97091a4" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 34565 } } }
I20260927 15:20:30.758826 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:30.759755 27302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:30.762419 27302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:30.767518 27302 catalog_manager.cc:1347] Generated new cluster ID: a6aa60846db9422db44e3e28d42aa085
I20260927 15:20:30.767580 27302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:30.817194 27302 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:30.818521 27302 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:30.827755 27302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4: Generated new TSK 0
I20260927 15:20:30.828487 27302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:20:30.864496 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34565
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:30.974575 27314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:30.974835 27314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:30.974877 27314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:30.978842 27314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:30.978914 27314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:30.978989 27314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:20:30.984462 27314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:30.985464 27314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:30.986487 27314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:30.995662 27322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:30.995656 27319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:30.995647 27320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:30.995914 27314 server_base.cc:1034] running on GCE node
I20260927 15:20:30.996420 27314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:30.997042 27314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:30.998221 27314 hybrid_clock.cc:648] HybridClock initialized: now 1790522430998182 us; error 53 us; skew 500 ppm
Sep 27 15:20:30 dist-test-slave-m4lt krb5kdc[27207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:31.001441 27314 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:20:31.002419 27314 webserver.cc:460] Webserver started at http://127.9.242.129:42395/ using document root <none> and password file <none>
I20260927 15:20:31.002986 27314 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:31.003055 27314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:31.003274 27314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:31.005395 27314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "20a4c1c9f9cd4da6a0a130d331f827b7"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "c306678c762a06b01cd85f945c1e51a4"
server_key_iv: "1a3363f86304b3ddfec27b62ab09da86"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.005991 27314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "20a4c1c9f9cd4da6a0a130d331f827b7"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "c306678c762a06b01cd85f945c1e51a4"
server_key_iv: "1a3363f86304b3ddfec27b62ab09da86"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.009915 27314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:20:31.013041 27329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.014489 27314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:31.014596 27314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "20a4c1c9f9cd4da6a0a130d331f827b7"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "c306678c762a06b01cd85f945c1e51a4"
server_key_iv: "1a3363f86304b3ddfec27b62ab09da86"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.014703 27314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:31.046586 27314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:31.049543 27314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:31.049733 27314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:31.050320 27314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:31.051141 27314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:31.051186 27314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.051225 27314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:31.051254 27314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.059654 27314 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:35993
I20260927 15:20:31.059676 27442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:35993 every 8 connection(s)
I20260927 15:20:31.060505 27314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
Sep 27 15:20:31 dist-test-slave-m4lt krb5kdc[27207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:31.070741 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27314
I20260927 15:20:31.070876 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:31.071208 10186 external_mini_cluster.cc:1428] Setting key e92c4da65c002c9a36f275be76347b8e
I20260927 15:20:31.074653 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.062329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:50589 (local address 127.9.242.190:34565)
0927 15:20:31.062491 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.062494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.063326 (+ 832us) server_negotiation.cc:407] Connection header received
0927 15:20:31.064673 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.064676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.064765 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.064846 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:31.066159 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.066694 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.067864 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.068093 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.068183 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.070796 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:31.070800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:31.070804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:31.070836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:31.072570 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.073083 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.073087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.073089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.073172 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.073532 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.073536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.073537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.073768 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:31.073861 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.074365 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.074490 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:20:31.075490 27445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.062696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34565 (local address 127.9.242.129:50589)
0927 15:20:31.063118 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.063160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.064457 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.065047 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.065054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.065458 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:31.066010 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.066025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.066826 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.066829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.067740 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.067749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.068251 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.068256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.068431 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.068999 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:31.069016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:31.070594 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:31.072679 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:31.072685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:31.072701 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:31.072986 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:31.073252 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:31.073255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:31.073256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:31.073441 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:31.073861 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:31.073868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:31.074132 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.074309 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.074654 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
I20260927 15:20:31.076745 27443 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34565
I20260927 15:20:31.076970 27443 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:31.077498 27443 heartbeater.cc:511] Master 127.9.242.190:34565 requested a full tablet report, sending...
I20260927 15:20:31.079211 27252 ts_manager.cc:194] Registered new tserver with Master: 20a4c1c9f9cd4da6a0a130d331f827b7 (127.9.242.129:35993)
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:20:31.080745 27252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:50589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:20:31.097702 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34565
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:31.204336 27448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:31.204593 27448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:31.204633 27448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:31.208612 27448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:31.208673 27448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:31.208730 27448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:20:31.213929 27448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:31.214900 27448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:31.215839 27448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:20:31.225843 27448 server_base.cc:1034] running on GCE node
W20260927 15:20:31.225786 27456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:31.225790 27453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:31.226074 27454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:31.226558 27448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:31.227320 27448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:31.228504 27448 hybrid_clock.cc:648] HybridClock initialized: now 1790522431228470 us; error 52 us; skew 500 ppm
Sep 27 15:20:31 dist-test-slave-m4lt krb5kdc[27207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522431, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:31.231429 27448 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:20:31.232391 27448 webserver.cc:460] Webserver started at http://127.9.242.130:35977/ using document root <none> and password file <none>
I20260927 15:20:31.232982 27448 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:31.233042 27448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:31.233224 27448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:31.235363 27448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "82f9c199231a44a5bc62bbeac1f9610c"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "8d53e532881c4517a1b9e329b85d77f7"
server_key_iv: "707ec1434ce900250bb2d5b1de728f23"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.235893 27448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "82f9c199231a44a5bc62bbeac1f9610c"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "8d53e532881c4517a1b9e329b85d77f7"
server_key_iv: "707ec1434ce900250bb2d5b1de728f23"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.239781 27448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:20:31.242257 27463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.243539 27448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:31.243650 27448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "82f9c199231a44a5bc62bbeac1f9610c"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "8d53e532881c4517a1b9e329b85d77f7"
server_key_iv: "707ec1434ce900250bb2d5b1de728f23"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.243764 27448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:31.263777 27448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:31.266912 27448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:31.267086 27448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:31.267683 27448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:31.268501 27448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:31.268553 27448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.268606 27448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:31.268635 27448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.277009 27448 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:43155
I20260927 15:20:31.277037 27576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:43155 every 8 connection(s)
I20260927 15:20:31.277887 27448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:31.283722 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27448
I20260927 15:20:31.283862 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:31.284147 10186 external_mini_cluster.cc:1428] Setting key a779cf18a2366f3d8b93c90392775ddd
Sep 27 15:20:31 dist-test-slave-m4lt krb5kdc[27207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522431, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:31.292171 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.279592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:47539 (local address 127.9.242.190:34565)
0927 15:20:31.279789 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.279794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.280693 (+ 899us) server_negotiation.cc:407] Connection header received
0927 15:20:31.281930 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.281932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.281977 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.282048 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:31.283169 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.283930 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.285173 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.285471 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.285575 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.288494 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:31.288497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:31.288499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:31.288524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:31.290169 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.290642 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.290646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.290648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.290737 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.291099 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.291102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.291104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.291324 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:31.291428 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.291882 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.292007 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 15:20:31.292991 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.280044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34565 (local address 127.9.242.130:47539)
0927 15:20:31.280536 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.280579 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.281750 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.282190 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.282196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.282562 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:31.283024 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.283037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.284044 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.284046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.285054 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.285062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.285631 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.285635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.285844 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.286518 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:31.286538 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:31.288294 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:31.290285 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:31.290290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:31.290302 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:31.290549 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:31.290851 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:31.290860 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:31.290861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:31.291006 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:31.291421 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:31.291426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:31.291654 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.291823 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.292084 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:20:31.294229 27577 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34565
I20260927 15:20:31.294477 27577 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:31.294942 27577 heartbeater.cc:511] Master 127.9.242.190:34565 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:20:31.296057 27252 ts_manager.cc:194] Registered new tserver with Master: 82f9c199231a44a5bc62bbeac1f9610c (127.9.242.130:43155)
I20260927 15:20:31.296663 27252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:47539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:20:31.310799 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34565
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:31.420082 27582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:31.420328 27582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:31.420372 27582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:31.424407 27582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:31.424464 27582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:31.424520 27582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:20:31.429836 27582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:34565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:31.430840 27582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:31.431828 27582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:31.441692 27587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:31.441772 27582 server_base.cc:1034] running on GCE node
W20260927 15:20:31.441709 27588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:31.441910 27590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:31.442317 27582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:31.442956 27582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:31.444144 27582 hybrid_clock.cc:648] HybridClock initialized: now 1790522431444122 us; error 35 us; skew 500 ppm
Sep 27 15:20:31 dist-test-slave-m4lt krb5kdc[27207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522431, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:31.446841 27582 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:20:31.447728 27582 webserver.cc:460] Webserver started at http://127.9.242.131:42259/ using document root <none> and password file <none>
I20260927 15:20:31.448256 27582 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:31.448325 27582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:31.448523 27582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:31.450625 27582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "4e2c50e29e594008a823c39460bc4e7e"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "8df43fc0276278fc8303c184091b8f86"
server_key_iv: "fff33b2e5e4f166dade475f2f4013266"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.451143 27582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "4e2c50e29e594008a823c39460bc4e7e"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "8df43fc0276278fc8303c184091b8f86"
server_key_iv: "fff33b2e5e4f166dade475f2f4013266"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.454672 27582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:31.456902 27597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.458196 27582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:31.458298 27582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "4e2c50e29e594008a823c39460bc4e7e"
format_stamp: "Formatted at 2026-09-27 15:20:31 on dist-test-slave-m4lt"
server_key: "8df43fc0276278fc8303c184091b8f86"
server_key_iv: "fff33b2e5e4f166dade475f2f4013266"
server_key_version: "encryptionkey@0"
I20260927 15:20:31.458427 27582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:31.470162 27582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:31.473097 27582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:31.473271 27582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:31.473862 27582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:31.474634 27582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:31.474678 27582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.474730 27582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:31.474756 27582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:31.483071 27582 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:45825
I20260927 15:20:31.483093 27710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:45825 every 8 connection(s)
I20260927 15:20:31.483937 27582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:31.486830 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27582
I20260927 15:20:31.486955 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:31.487198 10186 external_mini_cluster.cc:1428] Setting key a7de15ea0d4852d6a929ebae2331a5ac
Sep 27 15:20:31 dist-test-slave-m4lt krb5kdc[27207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522431, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:31.496958 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.485607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:48823 (local address 127.9.242.190:34565)
0927 15:20:31.485793 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.485797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.486704 (+ 907us) server_negotiation.cc:407] Connection header received
0927 15:20:31.488159 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.488162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.488215 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.488287 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:31.489602 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.490200 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.491292 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.491514 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.491598 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.493842 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:31.493844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:31.493846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:31.493869 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:31.495110 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.495573 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.495576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.495578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.495649 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.495938 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.495941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.495943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.496121 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:31.496201 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.496674 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.496788 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 15:20:31.497747 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.486115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:34565 (local address 127.9.242.131:48823)
0927 15:20:31.486557 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.486599 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.487966 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.488471 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.488478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.488928 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:31.489471 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.489484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.490311 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.490315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.491186 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.491195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.491638 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.491642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.491806 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.492322 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:31.492337 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:31.493693 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:31.495213 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:31.495219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:31.495234 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:31.495486 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:31.495724 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:31.495727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:31.495729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:31.495858 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:31.496212 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:31.496219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:31.496462 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.496619 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.496925 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260927 15:20:31.498772 27711 heartbeater.cc:347] Connected to a master server at 127.9.242.190:34565
I20260927 15:20:31.498984 27711 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:31.499408 27711 heartbeater.cc:511] Master 127.9.242.190:34565 requested a full tablet report, sending...
I20260927 15:20:31.500303 27252 ts_manager.cc:194] Registered new tserver with Master: 4e2c50e29e594008a823c39460bc4e7e (127.9.242.131:45825)
I20260927 15:20:31.500981 27252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:48823
I20260927 15:20:31.501181 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:20:31.510118 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.503150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.9.242.190:34565)
0927 15:20:31.503266 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.503271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.503500 (+ 229us) server_negotiation.cc:407] Connection header received
0927 15:20:31.503621 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.503624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.503663 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.503724 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:31.504375 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.504901 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.505819 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.506031 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.506111 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.507379 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:31.507383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:31.507386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:31.507408 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:31.508607 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.508972 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.508976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.508977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.509044 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:31.509279 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:31.509282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:31.509284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:31.509483 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:31.509594 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.509875 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.509978 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 15:20:31.512740 27252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:31.516018 27252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:31.529803 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.524251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.9.242.130:43155)
0927 15:20:31.524603 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.524614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.524843 (+ 229us) server_negotiation.cc:407] Connection header received
0927 15:20:31.524912 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.524916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.525102 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.525246 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:31.525604 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.526842 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.528211 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.529039 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.529144 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.529289 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.529562 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.529668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260927 15:20:31.529801 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.524569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35993 (local address 127.0.0.1:50484)
0927 15:20:31.525151 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.525163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.525256 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.525925 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.525928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.525944 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:31.526028 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.526034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.526961 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.526964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.528387 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.528394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.529213 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.529216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.529326 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.529345 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.529437 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.529606 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":150,"thread_start_us":84,"threads_started":1}
I20260927 15:20:31.529845 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.523993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.9.242.131:45825)
0927 15:20:31.524793 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.524805 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.524826 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:20:31.524900 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.524905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.525103 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.525577 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:31.525742 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.526842 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.528469 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.529225 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.529358 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.529481 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.529600 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.529716 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":148,"threads_started":1}
I20260927 15:20:31.529803 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.523869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:45825 (local address 127.0.0.1:46516)
0927 15:20:31.524530 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.524578 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.524694 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.525270 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.525276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.525299 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:31.525433 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.525440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.527005 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.527008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.528349 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.528360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.529358 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.529361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.529444 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.529465 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.529551 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.529622 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":60,"thread_start_us":69,"threads_started":1}
I20260927 15:20:31.530489 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.523919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:43155 (local address 127.0.0.1:52954)
0927 15:20:31.524605 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.524618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.524711 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.525395 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.525398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.525413 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:31.525496 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.525501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.526986 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.526990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.528091 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.528100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.529157 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.529160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.529281 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.529344 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.529442 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.530402 (+ 960us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":920,"thread_start_us":100,"threads_started":1}
I20260927 15:20:31.530571 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.524952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.9.242.129:35993)
0927 15:20:31.525479 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.525494 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.525514 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:20:31.525583 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.525588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.525803 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.525928 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:31.526125 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.526847 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.528504 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.529096 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.530175 (+ 1079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.530322 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.530365 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.530446 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":101,"threads_started":1}
I20260927 15:20:31.532992 27374 tablet_service.cc:1467] Processing CreateTablet for tablet 964f2055b0be49058c2f26b15ce1cb99 (DEFAULT_TABLE table=test-table [id=d46715dfff7644018bc84fba3131670e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:31.533057 27511 tablet_service.cc:1467] Processing CreateTablet for tablet 964f2055b0be49058c2f26b15ce1cb99 (DEFAULT_TABLE table=test-table [id=d46715dfff7644018bc84fba3131670e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:31.533018 27645 tablet_service.cc:1467] Processing CreateTablet for tablet 964f2055b0be49058c2f26b15ce1cb99 (DEFAULT_TABLE table=test-table [id=d46715dfff7644018bc84fba3131670e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:31.534013 27374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964f2055b0be49058c2f26b15ce1cb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:31.534032 27511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964f2055b0be49058c2f26b15ce1cb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:31.534032 27645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964f2055b0be49058c2f26b15ce1cb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:31.541086 27728 tablet_bootstrap.cc:492] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: Bootstrap starting.
I20260927 15:20:31.541539 27729 tablet_bootstrap.cc:492] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: Bootstrap starting.
I20260927 15:20:31.541966 27730 tablet_bootstrap.cc:492] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Bootstrap starting.
I20260927 15:20:31.543182 27728 tablet_bootstrap.cc:654] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:31.543696 27729 tablet_bootstrap.cc:654] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:31.543840 27730 tablet_bootstrap.cc:654] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:31.544261 27728 log.cc:824] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:31.544974 27729 log.cc:824] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:31.544974 27730 log.cc:824] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:31.546344 27728 tablet_bootstrap.cc:492] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: No bootstrap required, opened a new log
I20260927 15:20:31.546548 27728 ts_tablet_manager.cc:1397] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 15:20:31.546909 27729 tablet_bootstrap.cc:492] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: No bootstrap required, opened a new log
I20260927 15:20:31.547063 27730 tablet_bootstrap.cc:492] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: No bootstrap required, opened a new log
I20260927 15:20:31.547091 27729 ts_tablet_manager.cc:1397] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:31.547220 27730 ts_tablet_manager.cc:1397] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:20:31.549702 27728 raft_consensus.cc:348] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.549932 27728 raft_consensus.cc:374] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:31.550007 27728 raft_consensus.cc:729] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a4c1c9f9cd4da6a0a130d331f827b7, State: Initialized, Role: FOLLOWER
I20260927 15:20:31.550422 27728 consensus_queue.cc:260] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.550876 27730 raft_consensus.cc:348] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.550876 27729 raft_consensus.cc:348] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.551115 27729 raft_consensus.cc:374] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:31.551115 27730 raft_consensus.cc:374] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:31.551168 27730 raft_consensus.cc:729] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2c50e29e594008a823c39460bc4e7e, State: Initialized, Role: FOLLOWER
I20260927 15:20:31.551218 27729 raft_consensus.cc:729] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f9c199231a44a5bc62bbeac1f9610c, State: Initialized, Role: FOLLOWER
I20260927 15:20:31.551270 27728 ts_tablet_manager.cc:1428] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 15:20:31.551617 27443 heartbeater.cc:503] Master 127.9.242.190:34565 was elected leader, sending a full tablet report...
I20260927 15:20:31.551666 27729 consensus_queue.cc:260] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.551653 27730 consensus_queue.cc:260] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.552484 27577 heartbeater.cc:503] Master 127.9.242.190:34565 was elected leader, sending a full tablet report...
I20260927 15:20:31.552584 27729 ts_tablet_manager.cc:1428] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 15:20:31.552584 27730 ts_tablet_manager.cc:1428] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:31.553435 27711 heartbeater.cc:503] Master 127.9.242.190:34565 was elected leader, sending a full tablet report...
W20260927 15:20:31.561587 27444 tablet.cc:2369] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:20:31.734786 27712 tablet.cc:2369] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:20:31.779029 27578 tablet.cc:2369] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:31.795295 27734 raft_consensus.cc:482] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:31.795542 27734 raft_consensus.cc:504] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.796892 27734 leader_election.cc:288] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e2c50e29e594008a823c39460bc4e7e (127.9.242.131:45825), 82f9c199231a44a5bc62bbeac1f9610c (127.9.242.130:43155)
I20260927 15:20:31.801819 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.797505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:57267 (local address 127.9.242.131:45825)
0927 15:20:31.797636 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.797640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.797676 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:20:31.797731 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.797734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.797781 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.797850 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:31.798194 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.798997 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.800763 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.801231 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.801338 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.801392 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.801582 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.801685 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:20:31.801896 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.797141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:43155 (local address 127.9.242.129:42319)
0927 15:20:31.797528 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.797539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.797677 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.797907 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.797909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.797926 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:31.798116 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.798123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.799123 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.799126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.800621 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.800631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.801507 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.801510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.801597 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.801616 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.801689 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.801757 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":187,"thread_start_us":90,"threads_started":1}
I20260927 15:20:31.802281 27725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.797230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:42319 (local address 127.9.242.130:43155)
0927 15:20:31.797365 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.797370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.797620 (+ 250us) server_negotiation.cc:407] Connection header received
0927 15:20:31.797766 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.797769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.797822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.797898 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:31.798265 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.798997 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.800742 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.801392 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.801523 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.801572 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.802093 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.802173 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:20:31.802659 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.797139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:45825 (local address 127.9.242.129:57267)
0927 15:20:31.797363 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.797377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:31.797481 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:31.797860 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:31.797863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:31.797881 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:31.798084 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.798097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.799123 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.799126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.800620 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:31.800631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.801341 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.801345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:31.801439 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.801462 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:20:31.802291 (+ 829us) client_negotiation.cc:240] Negotiation successful
0927 15:20:31.802515 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":135,"thread_start_us":65,"threads_started":1}
I20260927 15:20:31.802924 27531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f2055b0be49058c2f26b15ce1cb99" candidate_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f9c199231a44a5bc62bbeac1f9610c" is_pre_election: true
I20260927 15:20:31.803305 27531 raft_consensus.cc:2454] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a4c1c9f9cd4da6a0a130d331f827b7 in term 0.
I20260927 15:20:31.803399 27665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f2055b0be49058c2f26b15ce1cb99" candidate_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2c50e29e594008a823c39460bc4e7e" is_pre_election: true
I20260927 15:20:31.803706 27665 raft_consensus.cc:2454] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a4c1c9f9cd4da6a0a130d331f827b7 in term 0.
I20260927 15:20:31.803859 27331 leader_election.cc:302] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4c1c9f9cd4da6a0a130d331f827b7, 82f9c199231a44a5bc62bbeac1f9610c; no voters:
I20260927 15:20:31.804136 27734 raft_consensus.cc:2781] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:31.804242 27734 raft_consensus.cc:482] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:31.804301 27734 raft_consensus.cc:3037] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:31.805557 27734 raft_consensus.cc:504] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.805944 27734 leader_election.cc:288] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4e2c50e29e594008a823c39460bc4e7e (127.9.242.131:45825), 82f9c199231a44a5bc62bbeac1f9610c (127.9.242.130:43155)
I20260927 15:20:31.806342 27665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f2055b0be49058c2f26b15ce1cb99" candidate_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2c50e29e594008a823c39460bc4e7e"
I20260927 15:20:31.806447 27531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964f2055b0be49058c2f26b15ce1cb99" candidate_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f9c199231a44a5bc62bbeac1f9610c"
I20260927 15:20:31.806468 27665 raft_consensus.cc:3037] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:31.806558 27531 raft_consensus.cc:3037] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:31.807716 27665 raft_consensus.cc:2454] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a4c1c9f9cd4da6a0a130d331f827b7 in term 1.
I20260927 15:20:31.807716 27531 raft_consensus.cc:2454] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a4c1c9f9cd4da6a0a130d331f827b7 in term 1.
I20260927 15:20:31.808022 27331 leader_election.cc:302] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a4c1c9f9cd4da6a0a130d331f827b7, 82f9c199231a44a5bc62bbeac1f9610c; no voters:
I20260927 15:20:31.808229 27734 raft_consensus.cc:2781] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:31.808436 27734 raft_consensus.cc:686] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 1 LEADER]: Becoming Leader. State: Replica: 20a4c1c9f9cd4da6a0a130d331f827b7, State: Running, Role: LEADER
I20260927 15:20:31.808791 27734 consensus_queue.cc:237] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } }
I20260927 15:20:31.812068 27250 catalog_manager.cc:5495] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a4c1c9f9cd4da6a0a130d331f827b7 (127.9.242.129). New cstate: current_term: 1 leader_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a4c1c9f9cd4da6a0a130d331f827b7" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35993 } health_report { overall_health: HEALTHY } } }
I20260927 15:20:31.891994 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.888403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.9.242.129:35993)
0927 15:20:31.888529 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:31.888534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:31.888549 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:20:31.888668 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:31.888671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:31.888745 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:31.888836 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:31.889083 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.889610 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.890611 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:31.890836 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:31.890922 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:31.891210 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:31.891627 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:31.891744 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:31.891803 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 15:20:31.891860 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:20:31.897738 27665 raft_consensus.cc:1261] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 1 FOLLOWER]: Refusing update from remote peer 20a4c1c9f9cd4da6a0a130d331f827b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:31.897737 27531 raft_consensus.cc:1261] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 1 FOLLOWER]: Refusing update from remote peer 20a4c1c9f9cd4da6a0a130d331f827b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:31.898370 27739 consensus_queue.cc:1035] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2c50e29e594008a823c39460bc4e7e" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:31.898543 27734 consensus_queue.cc:1035] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f9c199231a44a5bc62bbeac1f9610c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:31.905573 27744 mvcc.cc:205] Tried to move back new op lower bound from 7333979881046048768 to 7333979880691613696. Current Snapshot: MvccSnapshot[applied={T|T < 7333979881046048768}]
I20260927 15:20:31.905620 27743 mvcc.cc:205] Tried to move back new op lower bound from 7333979881046048768 to 7333979880691613696. Current Snapshot: MvccSnapshot[applied={T|T < 7333979881046048768}]
I20260927 15:20:31.913131 27251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:31.913293 27251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:31.915706 27251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76f6eacf6aa046f281812856f97091a4: Sending DeleteTablet for 3 replicas of tablet 964f2055b0be49058c2f26b15ce1cb99
I20260927 15:20:31.916473 27511 tablet_service.cc:1514] Processing DeleteTablet for tablet 964f2055b0be49058c2f26b15ce1cb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:31 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:52954
I20260927 15:20:31.916592 27645 tablet_service.cc:1514] Processing DeleteTablet for tablet 964f2055b0be49058c2f26b15ce1cb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:31 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:46516
I20260927 15:20:31.916951 27752 tablet_replica.cc:330] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: stopping tablet replica
I20260927 15:20:31.916991 27374 tablet_service.cc:1514] Processing DeleteTablet for tablet 964f2055b0be49058c2f26b15ce1cb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:31 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:50484
I20260927 15:20:31.917276 27752 raft_consensus.cc:2229] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:31.917455 27754 tablet_replica.cc:330] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7: stopping tablet replica
I20260927 15:20:31.917624 27752 raft_consensus.cc:2258] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:31.917876 27754 raft_consensus.cc:2229] T 964f2055b0be49058c2f26b15ce1cb99 P 20a4c1c9f9cd4da6a0a130d331f827b7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:31.918043 27753 tablet_replica.cc:330] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: stopping tablet replica
I20260927 15:20:31.918147 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27314
I20260927 15:20:31.918844 27753 raft_consensus.cc:2229] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:31.919137 27753 raft_consensus.cc:2258] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:31.920578 27753 ts_tablet_manager.cc:1905] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:31.922961 27753 ts_tablet_manager.cc:1918] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:31.923046 27753 log.cc:1196] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/964f2055b0be49058c2f26b15ce1cb99
I20260927 15:20:31.923388 27753 ts_tablet_manager.cc:1939] T 964f2055b0be49058c2f26b15ce1cb99 P 4e2c50e29e594008a823c39460bc4e7e: Deleting consensus metadata
I20260927 15:20:31.924319 27752 ts_tablet_manager.cc:1905] T 964f2055b0be49058c2f26b15ce1cb99 P 82f9c199231a44a5bc62bbeac1f9610c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 15:20:31.924644 27238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:35993 (error 108)
I20260927 15:20:31.924822 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27448
I20260927 15:20:31.925212 27238 catalog_manager.cc:4841] TS 4e2c50e29e594008a823c39460bc4e7e (127.9.242.131:45825): tablet 964f2055b0be49058c2f26b15ce1cb99 (table test-table [id=d46715dfff7644018bc84fba3131670e]) successfully deleted
I20260927 15:20:31.926187 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.925913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35993 (local address 127.0.0.1:50500)
0927 15:20:31.926038 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.926103 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 15:20:31.926338 27238 catalog_manager.cc:4584] TS 20a4c1c9f9cd4da6a0a130d331f827b7 (127.9.242.129:35993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 964f2055b0be49058c2f26b15ce1cb99: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35993: connect: Connection refused (error 111)
I20260927 15:20:31.929489 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27582
I20260927 15:20:31.930589 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:31.929536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:43155 (local address 127.0.0.1:52968)
0927 15:20:31.930479 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:31.930526 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.130:43155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":870}
I20260927 15:20:31.934120 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27220
2026-09-27T15:20:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2449 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T15:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:31Z Disabled control of system clock
I20260927 15:20:31.961558 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39449
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:39449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:20:32.069530 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:32.069796 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:32.069829 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:32.073835 27759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:32.073894 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:32.073907 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:32.073922 27759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:32.073935 27759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:32.078715 27759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:39449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:32.079734 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:32.081048 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:32.089339 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.089339 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.089403 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:32.089532 27759 server_base.cc:1034] running on GCE node
I20260927 15:20:32.090175 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:32.091331 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:32.092504 27759 hybrid_clock.cc:648] HybridClock initialized: now 1790522432092488 us; error 33 us; skew 500 ppm
I20260927 15:20:32.094434 27759 webserver.cc:460] Webserver started at http://127.9.242.190:39179/ using document root <none> and password file <none>
I20260927 15:20:32.094991 27759 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:32.095057 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:32.095240 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:32.097256 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "d70be7ac03f241b886afe1259a336728"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "16da47b88fc52f453e0d4d056d797319"
server_key_iv: "7cf051302b0295207ad76b8ea379bd26"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.097810 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "d70be7ac03f241b886afe1259a336728"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "16da47b88fc52f453e0d4d056d797319"
server_key_iv: "7cf051302b0295207ad76b8ea379bd26"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.101256 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:32.103808 27773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.104969 27759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:32.105087 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "d70be7ac03f241b886afe1259a336728"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "16da47b88fc52f453e0d4d056d797319"
server_key_iv: "7cf051302b0295207ad76b8ea379bd26"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.105177 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:32.133741 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:32.134447 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:32.134615 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:32.140956 27759 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:39449
I20260927 15:20:32.140965 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:39449 every 8 connection(s)
I20260927 15:20:32.141920 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:32.144894 27826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:32.147054 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27759
I20260927 15:20:32.147189 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:32.147481 10186 external_mini_cluster.cc:1428] Setting key 3cf06d92a5ef056f1427672f47535933
I20260927 15:20:32.150987 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728: Bootstrap starting.
I20260927 15:20:32.153213 27826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:32.154003 27826 log.cc:824] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:32.156193 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728: No bootstrap required, opened a new log
I20260927 15:20:32.156973 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.148919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50288 (local address 127.9.242.190:39449)
0927 15:20:32.149565 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.149580 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.149617 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:20:32.150338 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.150356 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.150755 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.151079 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:32.151498 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.152465 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.153346 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.153914 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.154062 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.154923 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:32.154929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:32.154946 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:32.155417 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:32.155498 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:32.155526 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:32.155619 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.155723 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.156025 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260927 15:20:32.160131 27826 raft_consensus.cc:348] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70be7ac03f241b886afe1259a336728" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39449 } }
I20260927 15:20:32.160370 27826 raft_consensus.cc:374] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:32.160424 27826 raft_consensus.cc:729] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70be7ac03f241b886afe1259a336728, State: Initialized, Role: FOLLOWER
I20260927 15:20:32.160805 27826 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70be7ac03f241b886afe1259a336728" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39449 } }
I20260927 15:20:32.160923 27826 raft_consensus.cc:388] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:32.160965 27826 raft_consensus.cc:482] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:32.161031 27826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:32.161947 27826 raft_consensus.cc:504] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70be7ac03f241b886afe1259a336728" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39449 } }
I20260927 15:20:32.162294 27826 leader_election.cc:302] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70be7ac03f241b886afe1259a336728; no voters:
I20260927 15:20:32.162597 27826 leader_election.cc:288] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:32.162767 27831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:32.163025 27831 raft_consensus.cc:686] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [term 1 LEADER]: Becoming Leader. State: Replica: d70be7ac03f241b886afe1259a336728, State: Running, Role: LEADER
I20260927 15:20:32.163348 27831 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70be7ac03f241b886afe1259a336728" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39449 } }
I20260927 15:20:32.163702 27826 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:32.164824 27832 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70be7ac03f241b886afe1259a336728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70be7ac03f241b886afe1259a336728" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39449 } } }
I20260927 15:20:32.164958 27832 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:32.165249 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70be7ac03f241b886afe1259a336728. Latest consensus state: current_term: 1 leader_uuid: "d70be7ac03f241b886afe1259a336728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70be7ac03f241b886afe1259a336728" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39449 } } }
I20260927 15:20:32.165339 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:32.165426 27840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:32.168027 27840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:32.173023 27840 catalog_manager.cc:1347] Generated new cluster ID: 2e86ae27649b4785a15f24ee750c9ce0
I20260927 15:20:32.173075 27840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:32.183881 27840 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:32.184783 27840 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:32.192472 27840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728: Generated new TSK 0
I20260927 15:20:32.193131 27840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:32.198143 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39449
--builtin_ntp_servers=127.9.242.148:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:20:32.306442 27850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:32.306686 27850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:32.306725 27850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:32.310808 27850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:32.310870 27850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:32.310938 27850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:20:32.315629 27850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:32.316664 27850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:32.318028 27850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:32.327571 27856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:32.327643 27850 server_base.cc:1034] running on GCE node
W20260927 15:20:32.327576 27855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.327576 27858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:32.328297 27850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:32.328938 27850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:32.330093 27850 hybrid_clock.cc:648] HybridClock initialized: now 1790522432330074 us; error 33 us; skew 500 ppm
I20260927 15:20:32.332048 27850 webserver.cc:460] Webserver started at http://127.9.242.129:43297/ using document root <none> and password file <none>
I20260927 15:20:32.332613 27850 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:32.332674 27850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:32.332866 27850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:32.334934 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "50a730f247374585ba52731b51717339"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "f0c9cf347c011b1f4a0b43761e1c38e8"
server_key_iv: "a6d17207d563aec1bb26442942ef2a36"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.335456 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "50a730f247374585ba52731b51717339"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "f0c9cf347c011b1f4a0b43761e1c38e8"
server_key_iv: "a6d17207d563aec1bb26442942ef2a36"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.338982 27850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:32.341310 27864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.342608 27850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:32.342707 27850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "50a730f247374585ba52731b51717339"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "f0c9cf347c011b1f4a0b43761e1c38e8"
server_key_iv: "a6d17207d563aec1bb26442942ef2a36"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.342842 27850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:32.355129 27850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:32.355755 27850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:32.355911 27850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:32.356451 27850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:32.357337 27850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:32.357384 27850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.357424 27850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:32.357451 27850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.366022 27850 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:45721
I20260927 15:20:32.366043 27977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:45721 every 8 connection(s)
I20260927 15:20:32.366954 27850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:32.374074 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27850
I20260927 15:20:32.374178 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:32.374462 10186 external_mini_cluster.cc:1428] Setting key dae3e51e562b31356021695c343612c2
I20260927 15:20:32.376235 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.368816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:47213 (local address 127.9.242.190:39449)
0927 15:20:32.369001 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.369006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.370102 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 15:20:32.371436 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.371438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.371486 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.371560 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:32.372530 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.373064 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.374308 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.374531 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.374631 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.375768 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:32.375772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:32.375774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:32.375788 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:32.375811 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:32.375820 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:32.375888 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.376033 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.376093 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 15:20:32.376475 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39449
--builtin_ntp_servers=127.9.242.148:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:20:32.377170 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.369262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39449 (local address 127.9.242.129:47213)
0927 15:20:32.369956 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.369998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.371218 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.371717 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.371724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.371809 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:32.372390 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.372407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.373169 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.373172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.374185 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.374197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.374699 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.374703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.374914 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.375493 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:32.375495 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:32.375543 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:32.375612 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:32.375895 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:32.375903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:32.375908 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.376014 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.376243 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260927 15:20:32.378342 27978 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39449
I20260927 15:20:32.378602 27978 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:32.379199 27978 heartbeater.cc:511] Master 127.9.242.190:39449 requested a full tablet report, sending...
I20260927 15:20:32.380793 27786 ts_manager.cc:194] Registered new tserver with Master: 50a730f247374585ba52731b51717339 (127.9.242.129:45721)
I20260927 15:20:32.381966 27786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.129:47213
W20260927 15:20:32.485383 27981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:32.485674 27981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:32.485735 27981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:32.489778 27981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:32.489830 27981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:32.489894 27981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:20:32.494699 27981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:32.495785 27981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:32.497063 27981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:32.506191 27989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.506191 27987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:32.506263 27981 server_base.cc:1034] running on GCE node
W20260927 15:20:32.506219 27986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:32.506805 27981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:32.507403 27981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:32.508566 27981 hybrid_clock.cc:648] HybridClock initialized: now 1790522432508545 us; error 36 us; skew 500 ppm
I20260927 15:20:32.510352 27981 webserver.cc:460] Webserver started at http://127.9.242.130:35365/ using document root <none> and password file <none>
I20260927 15:20:32.510905 27981 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:32.510969 27981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:32.511147 27981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:32.513149 27981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "12855b8a7e864d13a96494ccafce07db"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "bfe08d7f032927d9bcd14ed6912f7e0b"
server_key_iv: "a5d0bdd3f4429b2fe26713b04685a54f"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.513695 27981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "12855b8a7e864d13a96494ccafce07db"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "bfe08d7f032927d9bcd14ed6912f7e0b"
server_key_iv: "a5d0bdd3f4429b2fe26713b04685a54f"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.517174 27981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:32.519529 27995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.520653 27981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:32.520754 27981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "12855b8a7e864d13a96494ccafce07db"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "bfe08d7f032927d9bcd14ed6912f7e0b"
server_key_iv: "a5d0bdd3f4429b2fe26713b04685a54f"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.520853 27981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:32.544580 27981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:32.545312 27981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:32.545507 27981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:32.546144 27981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:32.547118 27981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:32.547163 27981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.547219 27981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:32.547246 27981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.555619 27981 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:45955
I20260927 15:20:32.555689 28108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:45955 every 8 connection(s)
I20260927 15:20:32.556520 27981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:32.562804 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 27981
I20260927 15:20:32.562914 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:32.563174 10186 external_mini_cluster.cc:1428] Setting key 95caa75529030df396fb64fcbb055421
I20260927 15:20:32.565333 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39449
--builtin_ntp_servers=127.9.242.148:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:20:32.566179 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.558253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:54025 (local address 127.9.242.190:39449)
0927 15:20:32.558386 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.558389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.559618 (+ 1229us) server_negotiation.cc:407] Connection header received
0927 15:20:32.560981 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.560985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.561041 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.561118 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:32.562080 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.562647 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.564042 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.564414 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.564508 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.565781 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:32.565784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:32.565786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:32.565799 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:32.565817 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:32.565825 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:32.565887 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.566021 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.566072 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 15:20:32.567203 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.558719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39449 (local address 127.9.242.130:54025)
0927 15:20:32.559466 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.559507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.560757 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.561336 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.561343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.561427 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:32.561942 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.561955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.562778 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.562781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.563945 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.563955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.564646 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.564649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.564875 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.565484 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:32.565486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:32.565541 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:32.565619 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:32.565896 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:32.565903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:32.565908 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.566022 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.566320 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260927 15:20:32.568449 28109 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39449
I20260927 15:20:32.568662 28109 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:32.569183 28109 heartbeater.cc:511] Master 127.9.242.190:39449 requested a full tablet report, sending...
I20260927 15:20:32.570128 27786 ts_manager.cc:194] Registered new tserver with Master: 12855b8a7e864d13a96494ccafce07db (127.9.242.130:45955)
I20260927 15:20:32.570716 27786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.130:54025
W20260927 15:20:32.672556 28112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:32.672807 28112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:32.672844 28112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:32.677181 28112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:32.677253 28112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:32.677309 28112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:20:32.682224 28112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:39449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:32.683302 28112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:32.684592 28112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:20:32.693604 28112 server_base.cc:1034] running on GCE node
W20260927 15:20:32.693593 28117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.693819 28118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:32.693598 28120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:32.694343 28112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:32.694909 28112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:32.696060 28112 hybrid_clock.cc:648] HybridClock initialized: now 1790522432696044 us; error 28 us; skew 500 ppm
I20260927 15:20:32.697882 28112 webserver.cc:460] Webserver started at http://127.9.242.131:44275/ using document root <none> and password file <none>
I20260927 15:20:32.698428 28112 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:32.698486 28112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:32.698673 28112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:32.700641 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "d781b719916b897b73b2c7b98cab7556"
server_key_iv: "9f5a0cebcca573c43f84003e9b4dc3ea"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.701164 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "d781b719916b897b73b2c7b98cab7556"
server_key_iv: "9f5a0cebcca573c43f84003e9b4dc3ea"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.704702 28112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 15:20:32.707002 28126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.708173 28112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:20:32.708272 28112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf"
format_stamp: "Formatted at 2026-09-27 15:20:32 on dist-test-slave-m4lt"
server_key: "d781b719916b897b73b2c7b98cab7556"
server_key_iv: "9f5a0cebcca573c43f84003e9b4dc3ea"
server_key_version: "encryptionkey@0"
I20260927 15:20:32.708375 28112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:32.732836 28112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:32.733528 28112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:32.733719 28112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:32.734288 28112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:32.735209 28112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:32.735255 28112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.735294 28112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:32.735321 28112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:32.743451 28112 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:35397
I20260927 15:20:32.743487 28239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:35397 every 8 connection(s)
I20260927 15:20:32.744308 28112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:32.751586 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 28112
I20260927 15:20:32.751695 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:32.751950 10186 external_mini_cluster.cc:1428] Setting key fdab9d33bb41a3515998ed93a6815f7c
I20260927 15:20:32.753784 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.746129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:59821 (local address 127.9.242.190:39449)
0927 15:20:32.746299 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.746304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.747488 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 15:20:32.748853 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.748857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.748913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.748984 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:32.749935 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.750519 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.751756 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.752001 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.752096 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.753262 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:32.753266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:32.753268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:32.753281 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:32.753304 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:32.753315 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:32.753380 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.753555 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.753605 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 15:20:32.754701 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.746565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:39449 (local address 127.9.242.131:59821)
0927 15:20:32.747346 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.747390 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.748642 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.749157 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.749165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.749258 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:32.749805 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.749820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.750640 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.750643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.751612 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.751623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.752151 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.752155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.752340 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.752973 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:32.752975 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:32.753030 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:32.753104 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:32.753402 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:32.753411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:32.753417 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.753552 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.753879 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260927 15:20:32.755681 28240 heartbeater.cc:347] Connected to a master server at 127.9.242.190:39449
I20260927 15:20:32.755898 28240 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:32.756382 28240 heartbeater.cc:511] Master 127.9.242.190:39449 requested a full tablet report, sending...
I20260927 15:20:32.757253 27786 ts_manager.cc:194] Registered new tserver with Master: a7c0ad3c7cce429bb1aab05072a7f5bf (127.9.242.131:35397)
I20260927 15:20:32.757833 27786 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.131:59821
I20260927 15:20:32.765897 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:20:32.771219 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.767714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.9.242.190:39449)
0927 15:20:32.767846 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.767850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.767988 (+ 138us) server_negotiation.cc:407] Connection header received
0927 15:20:32.768107 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.768110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.768154 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.768216 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:32.768503 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.769011 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.769866 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.770085 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.770165 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.770846 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:32.770850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:32.770852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:32.770865 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:32.770882 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:32.770890 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:32.770944 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.771034 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.771088 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:20:32.773893 27786 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:32.776853 27786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:32.790151 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.784865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:35397 (local address 127.0.0.1:42938)
0927 15:20:32.785629 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.785641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.785744 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.786480 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.786486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.786505 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:32.786629 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.786635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.787742 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.787745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.788855 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.788864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.789611 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.789614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.789743 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.789785 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.789899 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.790009 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":94,"threads_started":1}
I20260927 15:20:32.790191 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.785098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.9.242.131:35397)
0927 15:20:32.785915 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.785928 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.785953 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:20:32.786029 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.786035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.786246 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.786397 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:32.786758 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.787644 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.788965 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.789495 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.789588 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.789813 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.789934 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.790029 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":137,"threads_started":1}
I20260927 15:20:32.790798 28251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.784803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:45721 (local address 127.0.0.1:47702)
0927 15:20:32.785139 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.785190 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.785315 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.786686 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.786689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.786705 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:32.786792 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.786798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.788215 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.788218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.789406 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.789415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.790451 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.790455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.790542 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.790565 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.790612 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.790676 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260927 15:20:32.791021 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.785539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:45955 (local address 127.0.0.1:57888)
0927 15:20:32.786262 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.786274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.786361 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.786938 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.786941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.786957 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:32.787039 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.787046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.788388 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.788399 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.789488 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.789495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.790687 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.790690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.790754 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.790771 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.790833 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.790902 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":70,"threads_started":1}
I20260927 15:20:32.791523 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.785720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.9.242.129:45721)
0927 15:20:32.786265 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.786283 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.786314 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:20:32.786385 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.786390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.786557 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.786684 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:32.786909 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.788071 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.789529 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.790301 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.791042 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.791219 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.791272 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.791372 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":96,"threads_started":1}
I20260927 15:20:32.793416 28174 tablet_service.cc:1467] Processing CreateTablet for tablet e165a6deb9924d8f937751f5ce6cc3e6 (DEFAULT_TABLE table=test-table [id=a5ecdcdc8d3d41d5a60099a1d1004eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:32.794229 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.785926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.9.242.130:45955)
0927 15:20:32.786196 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.786207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.786356 (+ 149us) server_negotiation.cc:407] Connection header received
0927 15:20:32.786464 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.786468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.786651 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.786785 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:32.787168 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.788279 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.789575 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.790364 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.790453 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.790619 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.790892 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.790988 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260927 15:20:32.794611 28174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e165a6deb9924d8f937751f5ce6cc3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:32.794241 27912 tablet_service.cc:1467] Processing CreateTablet for tablet e165a6deb9924d8f937751f5ce6cc3e6 (DEFAULT_TABLE table=test-table [id=a5ecdcdc8d3d41d5a60099a1d1004eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:32.795234 27912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e165a6deb9924d8f937751f5ce6cc3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:32.796581 28043 tablet_service.cc:1467] Processing CreateTablet for tablet e165a6deb9924d8f937751f5ce6cc3e6 (DEFAULT_TABLE table=test-table [id=a5ecdcdc8d3d41d5a60099a1d1004eba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:32.797297 28043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e165a6deb9924d8f937751f5ce6cc3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:32.801929 28257 tablet_bootstrap.cc:492] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Bootstrap starting.
I20260927 15:20:32.802356 28258 tablet_bootstrap.cc:492] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: Bootstrap starting.
I20260927 15:20:32.803169 28259 tablet_bootstrap.cc:492] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Bootstrap starting.
I20260927 15:20:32.804581 28258 tablet_bootstrap.cc:654] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:32.804718 28257 tablet_bootstrap.cc:654] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:32.805063 28259 tablet_bootstrap.cc:654] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:32.805399 28258 log.cc:824] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:32.805540 28257 log.cc:824] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:32.805801 28259 log.cc:824] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:32.807443 28258 tablet_bootstrap.cc:492] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: No bootstrap required, opened a new log
I20260927 15:20:32.807489 28259 tablet_bootstrap.cc:492] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: No bootstrap required, opened a new log
I20260927 15:20:32.807488 28257 tablet_bootstrap.cc:492] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: No bootstrap required, opened a new log
I20260927 15:20:32.807621 28258 ts_tablet_manager.cc:1397] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:20:32.807657 28259 ts_tablet_manager.cc:1397] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 15:20:32.807657 28257 ts_tablet_manager.cc:1397] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:32.811039 28258 raft_consensus.cc:348] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.811259 28258 raft_consensus.cc:374] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:32.811379 28258 raft_consensus.cc:729] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a730f247374585ba52731b51717339, State: Initialized, Role: FOLLOWER
I20260927 15:20:32.811805 28258 consensus_queue.cc:260] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.811702 28259 raft_consensus.cc:348] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.811713 28257 raft_consensus.cc:348] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.811939 28259 raft_consensus.cc:374] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:32.811939 28257 raft_consensus.cc:374] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:32.811992 28259 raft_consensus.cc:729] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12855b8a7e864d13a96494ccafce07db, State: Initialized, Role: FOLLOWER
I20260927 15:20:32.811992 28257 raft_consensus.cc:729] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c0ad3c7cce429bb1aab05072a7f5bf, State: Initialized, Role: FOLLOWER
I20260927 15:20:32.812407 28259 consensus_queue.cc:260] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.812407 28257 consensus_queue.cc:260] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.812831 28258 ts_tablet_manager.cc:1428] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:32.813172 27978 heartbeater.cc:503] Master 127.9.242.190:39449 was elected leader, sending a full tablet report...
I20260927 15:20:32.813309 28257 ts_tablet_manager.cc:1428] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:20:32.813319 28259 ts_tablet_manager.cc:1428] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:20:32.813632 28109 heartbeater.cc:503] Master 127.9.242.190:39449 was elected leader, sending a full tablet report...
I20260927 15:20:32.813879 28240 heartbeater.cc:503] Master 127.9.242.190:39449 was elected leader, sending a full tablet report...
W20260927 15:20:32.868103 27979 tablet.cc:2369] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:32.957029 28265 raft_consensus.cc:482] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:32.957245 28265 raft_consensus.cc:504] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.958489 28265 leader_election.cc:288] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50a730f247374585ba52731b51717339 (127.9.242.129:45721), 12855b8a7e864d13a96494ccafce07db (127.9.242.130:45955)
I20260927 15:20:32.962585 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.958842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:35167 (local address 127.9.242.129:45721)
0927 15:20:32.958967 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.958971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.958991 (+ 20us) server_negotiation.cc:407] Connection header received
0927 15:20:32.959094 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.959097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.959150 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.959599 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:32.959702 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.960486 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.961576 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.962081 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.962195 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.962255 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.962398 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.962464 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:20:32.962596 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.958758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:45721 (local address 127.9.242.131:35167)
0927 15:20:32.958877 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.958893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.958985 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.959289 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.959292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.959324 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:32.959485 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.959494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.960607 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.960610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.961439 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.961447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.962191 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.962195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.962292 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.962310 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.962375 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.962446 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"spinlock_wait_cycles":25856}
I20260927 15:20:32.962656 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.959144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:37243 (local address 127.9.242.130:45955)
0927 15:20:32.959260 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:32.959265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:32.959362 (+ 97us) server_negotiation.cc:407] Connection header received
0927 15:20:32.959471 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:32.959473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:32.959523 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:32.959597 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:32.959910 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.960619 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.961638 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.962136 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.962242 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.962299 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:32.962468 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 15:20:32.962539 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:20:32.962690 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:32.959023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:45955 (local address 127.9.242.131:37243)
0927 15:20:32.959291 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:32.959302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:32.959390 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:32.959642 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:32.959645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:32.959663 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:32.959790 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.959795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.960721 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.960724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.961513 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:32.961521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:32.962255 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:32.962259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:32.962357 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:32.962378 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:20:32.962441 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:20:32.962506 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"spinlock_wait_cycles":896,"thread_start_us":104,"threads_started":1}
I20260927 15:20:32.963785 27932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165a6deb9924d8f937751f5ce6cc3e6" candidate_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a730f247374585ba52731b51717339" is_pre_election: true
I20260927 15:20:32.963846 28063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165a6deb9924d8f937751f5ce6cc3e6" candidate_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12855b8a7e864d13a96494ccafce07db" is_pre_election: true
I20260927 15:20:32.964109 27932 raft_consensus.cc:2454] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c0ad3c7cce429bb1aab05072a7f5bf in term 0.
I20260927 15:20:32.964109 28063 raft_consensus.cc:2454] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7c0ad3c7cce429bb1aab05072a7f5bf in term 0.
I20260927 15:20:32.964605 28127 leader_election.cc:302] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12855b8a7e864d13a96494ccafce07db, a7c0ad3c7cce429bb1aab05072a7f5bf; no voters:
I20260927 15:20:32.964852 28265 raft_consensus.cc:2781] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:32.964928 28265 raft_consensus.cc:482] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:32.964962 28265 raft_consensus.cc:3037] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:32.966033 28265 raft_consensus.cc:504] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.966377 28265 leader_election.cc:288] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [CANDIDATE]: Term 1 election: Requested vote from peers 50a730f247374585ba52731b51717339 (127.9.242.129:45721), 12855b8a7e864d13a96494ccafce07db (127.9.242.130:45955)
I20260927 15:20:32.966780 27932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165a6deb9924d8f937751f5ce6cc3e6" candidate_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a730f247374585ba52731b51717339"
I20260927 15:20:32.966828 28063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e165a6deb9924d8f937751f5ce6cc3e6" candidate_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12855b8a7e864d13a96494ccafce07db"
I20260927 15:20:32.966928 28063 raft_consensus.cc:3037] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:32.966915 27932 raft_consensus.cc:3037] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:32.968286 28063 raft_consensus.cc:2454] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c0ad3c7cce429bb1aab05072a7f5bf in term 1.
I20260927 15:20:32.968511 27932 raft_consensus.cc:2454] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c0ad3c7cce429bb1aab05072a7f5bf in term 1.
I20260927 15:20:32.968595 28127 leader_election.cc:302] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12855b8a7e864d13a96494ccafce07db, a7c0ad3c7cce429bb1aab05072a7f5bf; no voters:
I20260927 15:20:32.968757 28265 raft_consensus.cc:2781] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:32.969005 28265 raft_consensus.cc:686] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 1 LEADER]: Becoming Leader. State: Replica: a7c0ad3c7cce429bb1aab05072a7f5bf, State: Running, Role: LEADER
I20260927 15:20:32.969322 28265 consensus_queue.cc:237] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } }
I20260927 15:20:32.972572 27789 catalog_manager.cc:5495] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to a7c0ad3c7cce429bb1aab05072a7f5bf (127.9.242.131). New cstate: current_term: 1 leader_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c0ad3c7cce429bb1aab05072a7f5bf" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 } health_report { overall_health: UNKNOWN } } }
W20260927 15:20:32.995224 28241 tablet.cc:2369] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:33.024801 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:33.021042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.9.242.131:35397)
0927 15:20:33.021174 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:33.021177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:33.021191 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:20:33.021320 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:33.021322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:33.021384 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:33.021474 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:33.021736 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:33.022255 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:33.023227 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:33.023508 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:33.023598 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:33.023902 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:33.024445 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:33.024538 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:33.024633 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 15:20:33.024680 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:20:33.030928 27932 raft_consensus.cc:1261] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339 [term 1 FOLLOWER]: Refusing update from remote peer a7c0ad3c7cce429bb1aab05072a7f5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:33.030928 28063 raft_consensus.cc:1261] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 1 FOLLOWER]: Refusing update from remote peer a7c0ad3c7cce429bb1aab05072a7f5bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:33.031630 28267 consensus_queue.cc:1035] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a730f247374585ba52731b51717339" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:33.031862 28265 consensus_queue.cc:1035] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "12855b8a7e864d13a96494ccafce07db" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:33.039384 28272 mvcc.cc:205] Tried to move back new op lower bound from 7333979885687099392 to 7333979885445185536. Current Snapshot: MvccSnapshot[applied={T|T < 7333979885687099392}]
I20260927 15:20:33.041227 28271 mvcc.cc:205] Tried to move back new op lower bound from 7333979885687099392 to 7333979885445185536. Current Snapshot: MvccSnapshot[applied={T|T < 7333979885687099392}]
I20260927 15:20:33.047536 27786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:33.047679 27786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:33.049988 27786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d70be7ac03f241b886afe1259a336728: Sending DeleteTablet for 3 replicas of tablet e165a6deb9924d8f937751f5ce6cc3e6
I20260927 15:20:33.050716 28043 tablet_service.cc:1514] Processing DeleteTablet for tablet e165a6deb9924d8f937751f5ce6cc3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:33 UTC) from {username='slave'} at 127.0.0.1:57888
I20260927 15:20:33.050711 28174 tablet_service.cc:1514] Processing DeleteTablet for tablet e165a6deb9924d8f937751f5ce6cc3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:33 UTC) from {username='slave'} at 127.0.0.1:42938
I20260927 15:20:33.050709 27912 tablet_service.cc:1514] Processing DeleteTablet for tablet e165a6deb9924d8f937751f5ce6cc3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:33 UTC) from {username='slave'} at 127.0.0.1:47702
I20260927 15:20:33.051256 28281 tablet_replica.cc:330] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: stopping tablet replica
I20260927 15:20:33.051481 28279 tablet_replica.cc:330] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: stopping tablet replica
I20260927 15:20:33.051561 28281 raft_consensus.cc:2229] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:33.051743 28279 raft_consensus.cc:2229] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:33.051898 28281 raft_consensus.cc:2258] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:33.052019 28279 raft_consensus.cc:2258] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:33.052198 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27850
I20260927 15:20:33.052225 28280 tablet_replica.cc:330] T e165a6deb9924d8f937751f5ce6cc3e6 P 50a730f247374585ba52731b51717339: stopping tablet replica
I20260927 15:20:33.053527 28281 ts_tablet_manager.cc:1905] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:33.053526 28279 ts_tablet_manager.cc:1905] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:33.056085 28281 ts_tablet_manager.cc:1918] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:33.056106 28279 ts_tablet_manager.cc:1918] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:33.056171 28279 log.cc:1196] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/e165a6deb9924d8f937751f5ce6cc3e6
I20260927 15:20:33.056180 28281 log.cc:1196] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/e165a6deb9924d8f937751f5ce6cc3e6
I20260927 15:20:33.056463 28281 ts_tablet_manager.cc:1939] T e165a6deb9924d8f937751f5ce6cc3e6 P a7c0ad3c7cce429bb1aab05072a7f5bf: Deleting consensus metadata
I20260927 15:20:33.056463 28279 ts_tablet_manager.cc:1939] T e165a6deb9924d8f937751f5ce6cc3e6 P 12855b8a7e864d13a96494ccafce07db: Deleting consensus metadata
I20260927 15:20:33.057544 27774 catalog_manager.cc:4841] TS 12855b8a7e864d13a96494ccafce07db (127.9.242.130:45955): tablet e165a6deb9924d8f937751f5ce6cc3e6 (table test-table [id=a5ecdcdc8d3d41d5a60099a1d1004eba]) successfully deleted
I20260927 15:20:33.057596 27776 catalog_manager.cc:4841] TS a7c0ad3c7cce429bb1aab05072a7f5bf (127.9.242.131:35397): tablet e165a6deb9924d8f937751f5ce6cc3e6 (table test-table [id=a5ecdcdc8d3d41d5a60099a1d1004eba]) successfully deleted
W20260927 15:20:33.060782 27774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:45721 (error 108)
I20260927 15:20:33.061441 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27981
I20260927 15:20:33.062700 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:33.061511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:45721 (local address 127.0.0.1:47716)
0927 15:20:33.062541 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:33.062614 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:45721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260927 15:20:33.063020 27774 catalog_manager.cc:4584] TS 50a730f247374585ba52731b51717339 (127.9.242.129:45721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e165a6deb9924d8f937751f5ce6cc3e6: Network error: Client connection negotiation failed: client connection to 127.9.242.129:45721: connect: Connection refused (error 111)
I20260927 15:20:33.066498 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 28112
I20260927 15:20:33.070714 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 27759
2026-09-27T15:20:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1135 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:33 dist-test-slave-m4lt krb5kdc[28285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:33 dist-test-slave-m4lt krb5kdc[28285](info): set up 2 sockets
Sep 27 15:20:33 dist-test-slave-m4lt krb5kdc[28285](info): commencing operation
krb5kdc: starting...
W20260927 15:20:34.025514 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:20:34.142251 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43733
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:34.249872 28298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:34.250123 28298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:34.250161 28298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:34.254108 28298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:34.254168 28298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:34.254185 28298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:34.254199 28298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:34.254213 28298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:34.259374 28298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:34.260377 28298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:34.261337 28298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:34.269277 28306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.269438 28303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.269412 28304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:34.269596 28298 server_base.cc:1034] running on GCE node
I20260927 15:20:34.270258 28298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:34.271350 28298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:34.272508 28298 hybrid_clock.cc:648] HybridClock initialized: now 1790522434272485 us; error 37 us; skew 500 ppm
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:34.275293 28298 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:20:34.276139 28298 webserver.cc:460] Webserver started at http://127.9.242.190:38285/ using document root <none> and password file <none>
I20260927 15:20:34.276728 28298 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:34.276805 28298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:34.276997 28298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:34.279008 28298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "6ed5ccb40b9e4976a134010b4a0b90fc"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "96e0bd2f1e09b285f6df0c752192f302"
server_key_iv: "a199cfa30a988dbd04a73d9196468ff0"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.279523 28298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "6ed5ccb40b9e4976a134010b4a0b90fc"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "96e0bd2f1e09b285f6df0c752192f302"
server_key_iv: "a199cfa30a988dbd04a73d9196468ff0"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.283169 28298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:34.285599 28313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.286938 28298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:34.287043 28298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "6ed5ccb40b9e4976a134010b4a0b90fc"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "96e0bd2f1e09b285f6df0c752192f302"
server_key_iv: "a199cfa30a988dbd04a73d9196468ff0"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.287137 28298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:34.323930 28298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:34.327303 28298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:34.327466 28298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:34.334132 28298 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:43733
I20260927 15:20:34.334174 28365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:43733 every 8 connection(s)
I20260927 15:20:34.335124 28298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:34.338002 28366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:34.338652 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 28298
I20260927 15:20:34.338783 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:34.339089 10186 external_mini_cluster.cc:1428] Setting key bcca9705342398afdcf5265f0bb8d928
I20260927 15:20:34.344209 28366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc: Bootstrap starting.
I20260927 15:20:34.346750 28366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:34.347630 28366 log.cc:824] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:34.349604 28366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc: No bootstrap required, opened a new log
I20260927 15:20:34.352958 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:34.340450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.9.242.190:43733)
0927 15:20:34.340835 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:34.340851 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:34.340886 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:20:34.341509 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:34.341525 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:34.341905 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:34.342183 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:34.343024 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.344024 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.345190 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.345744 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.345887 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:34.348652 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:34.348657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:34.348670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:34.348695 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:34.350318 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.350728 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.350733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.350737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.350858 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.351112 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.351115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.351116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.351484 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:34.351657 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:34.351946 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 15:20:34.352222 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
I20260927 15:20:34.353030 28366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43733 } }
I20260927 15:20:34.353286 28366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:34.353336 28366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed5ccb40b9e4976a134010b4a0b90fc, State: Initialized, Role: FOLLOWER
I20260927 15:20:34.353751 28366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43733 } }
I20260927 15:20:34.353881 28366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:34.353950 28366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:34.354036 28366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:34.355141 28366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43733 } }
I20260927 15:20:34.355472 28366 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed5ccb40b9e4976a134010b4a0b90fc; no voters:
I20260927 15:20:34.355739 28366 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:34.355836 28371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:34.356029 28371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [term 1 LEADER]: Becoming Leader. State: Replica: 6ed5ccb40b9e4976a134010b4a0b90fc, State: Running, Role: LEADER
I20260927 15:20:34.356304 28371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43733 } }
I20260927 15:20:34.356830 28366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:34.357825 28372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43733 } } }
I20260927 15:20:34.357959 28372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:34.358235 28373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed5ccb40b9e4976a134010b4a0b90fc. Latest consensus state: current_term: 1 leader_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed5ccb40b9e4976a134010b4a0b90fc" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43733 } } }
I20260927 15:20:34.358309 28380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:34.358321 28373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:34.361469 28380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:34.367246 28380 catalog_manager.cc:1347] Generated new cluster ID: e2905efc79714d829fcadc11d2f4dfc8
I20260927 15:20:34.367326 28380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:34.382876 28380 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:34.383999 28380 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:34.391331 28380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc: Generated new TSK 0
I20260927 15:20:34.392227 28380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:20:34.422019 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43733
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:34.529760 28392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:34.530014 28392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:34.530072 28392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:34.534000 28392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:34.534061 28392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:34.534140 28392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:20:34.539613 28392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:34.540653 28392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:34.541688 28392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:20:34.551137 28392 server_base.cc:1034] running on GCE node
W20260927 15:20:34.551087 28397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.551087 28400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.551095 28398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:34.551757 28392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:34.552451 28392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:34.553619 28392 hybrid_clock.cc:648] HybridClock initialized: now 1790522434553594 us; error 49 us; skew 500 ppm
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:34.560580 28392 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:20:34.561442 28392 webserver.cc:460] Webserver started at http://127.9.242.129:45287/ using document root <none> and password file <none>
I20260927 15:20:34.562008 28392 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:34.562074 28392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:34.562273 28392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:34.564301 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "05030175aeb645398a0362c4a0012a3a"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "4fcf120b1547fba7a553b77157c4b74e"
server_key_iv: "37408535867381200904f5e9a0724ad3"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.564836 28392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "05030175aeb645398a0362c4a0012a3a"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "4fcf120b1547fba7a553b77157c4b74e"
server_key_iv: "37408535867381200904f5e9a0724ad3"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.568593 28392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 15:20:34.570941 28407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.572207 28392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:20:34.572309 28392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "05030175aeb645398a0362c4a0012a3a"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "4fcf120b1547fba7a553b77157c4b74e"
server_key_iv: "37408535867381200904f5e9a0724ad3"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.572453 28392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:34.607276 28392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:34.610605 28392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:34.610778 28392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:34.611382 28392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:34.612201 28392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:34.612254 28392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.612306 28392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:34.612340 28392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.620853 28392 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:32991
I20260927 15:20:34.620899 28520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:32991 every 8 connection(s)
I20260927 15:20:34.621739 28392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:34.628510 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 28392
I20260927 15:20:34.628628 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:34.628949 10186 external_mini_cluster.cc:1428] Setting key 65e538213f6dd18d8f799d5b7dee9d64
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:34.635489 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:34.623481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:33959 (local address 127.9.242.190:43733)
0927 15:20:34.623612 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:34.623615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:34.624458 (+ 843us) server_negotiation.cc:407] Connection header received
0927 15:20:34.625720 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:34.625723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:34.625778 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:34.625887 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:34.627065 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.627588 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.628685 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.629028 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.629139 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:34.631930 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:34.631934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:34.631937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:34.631965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:34.633304 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.633875 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.633877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.633879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.633949 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.634278 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.634281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.634283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.634517 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:34.634592 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:34.635220 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 15:20:34.635363 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:20:34.636570 28523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:34.623889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43733 (local address 127.9.242.129:33959)
0927 15:20:34.624314 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:34.624356 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:34.625514 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:34.626062 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:34.626068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:34.626455 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:34.626923 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:34.626937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.627703 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.627706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:34.628573 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:34.628583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.629169 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.629172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:34.629339 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:34.629948 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:34.629963 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:34.631729 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:34.633421 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:34.633427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:34.633442 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:34.633786 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:34.634021 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:34.634024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:34.634026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:34.634192 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:34.634617 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:34.634624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:34.634957 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 15:20:34.635165 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 15:20:34.635511 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
I20260927 15:20:34.637859 28521 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43733
I20260927 15:20:34.638077 28521 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
I20260927 15:20:34.638552 28521 heartbeater.cc:511] Master 127.9.242.190:43733 requested a full tablet report, sending...
I20260927 15:20:34.640049 28330 ts_manager.cc:194] Registered new tserver with Master: 05030175aeb645398a0362c4a0012a3a (127.9.242.129:32991)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:20:34.641239 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:33959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:20:34.658028 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43733
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:34.765168 28526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:34.765440 28526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:34.765503 28526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:34.770323 28526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:34.770380 28526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:34.770452 28526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:20:34.775843 28526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:34.776933 28526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:34.777963 28526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:34.787340 28532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.787375 28534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.787354 28531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:34.787626 28526 server_base.cc:1034] running on GCE node
I20260927 15:20:34.788031 28526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:34.788654 28526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:34.789841 28526 hybrid_clock.cc:648] HybridClock initialized: now 1790522434789795 us; error 34 us; skew 500 ppm
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:34.792502 28526 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:20:34.793402 28526 webserver.cc:460] Webserver started at http://127.9.242.130:34879/ using document root <none> and password file <none>
I20260927 15:20:34.793980 28526 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:34.794042 28526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:34.794207 28526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:34.796157 28526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "39399cb378b04d83a0557793d8fa2679"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "c7692cddc226cdb499a805e4bcec5764"
server_key_iv: "cabc4ff79d5c331108049a591daeaf4b"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.796649 28526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "39399cb378b04d83a0557793d8fa2679"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "c7692cddc226cdb499a805e4bcec5764"
server_key_iv: "cabc4ff79d5c331108049a591daeaf4b"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.800292 28526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 15:20:34.802724 28541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.804001 28526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:34.804104 28526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "39399cb378b04d83a0557793d8fa2679"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "c7692cddc226cdb499a805e4bcec5764"
server_key_iv: "cabc4ff79d5c331108049a591daeaf4b"
server_key_version: "encryptionkey@0"
I20260927 15:20:34.804211 28526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:34.815655 28526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:34.818759 28526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:34.818938 28526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:34.819540 28526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:34.820367 28526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:34.820421 28526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.820459 28526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:34.820487 28526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:34.829366 28526 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:40919
I20260927 15:20:34.829521 28654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:40919 every 8 connection(s)
I20260927 15:20:34.830380 28526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:34.833801 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 28526
I20260927 15:20:34.833912 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:34.834174 10186 external_mini_cluster.cc:1428] Setting key ed4306f7e80ce79eb3822fce96c67d4e
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:20:34.844331 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:34.832258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:45113 (local address 127.9.242.190:43733)
0927 15:20:34.832406 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:34.832412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:34.833379 (+ 967us) server_negotiation.cc:407] Connection header received
0927 15:20:34.834876 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:34.834880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:34.834935 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:34.835020 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:34.836377 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.836895 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.838079 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.838302 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.838387 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:34.840737 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:34.840740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:34.840742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:34.840767 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:34.842305 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.842852 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.842856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.842857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.842930 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:34.843250 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:34.843253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:34.843255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:34.843439 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:34.843524 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:34.844043 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 15:20:34.844165 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
I20260927 15:20:34.845216 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:34.832744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43733 (local address 127.9.242.130:45113)
0927 15:20:34.833225 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:34.833269 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:34.834656 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:34.835224 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:34.835233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:34.835723 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:34.836223 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:34.836238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.837009 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.837012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:34.837962 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:34.837971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:34.838444 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:34.838447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:34.838606 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:34.839161 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:34.839176 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:34.840574 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:34.842430 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:34.842436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:34.842452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:34.842757 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:34.843012 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:34.843015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:34.843017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:34.843155 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:34.843535 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:34.843542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:34.843803 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 15:20:34.843987 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 15:20:34.844345 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20260927 15:20:34.846349 28655 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43733
I20260927 15:20:34.846578 28655 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:34.847098 28655 heartbeater.cc:511] Master 127.9.242.190:43733 requested a full tablet report, sending...
I20260927 15:20:34.848115 28330 ts_manager.cc:194] Registered new tserver with Master: 39399cb378b04d83a0557793d8fa2679 (127.9.242.130:40919)
I20260927 15:20:34.848881 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:45113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:20:34.859901 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43733
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:34.970829 28660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:34.971096 28660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:34.971138 28660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:34.975106 28660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:34.975163 28660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:34.975238 28660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:20:34.980578 28660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:43733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:34.981669 28660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:34.982671 28660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:34.992442 28668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.992448 28665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:34.992442 28666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:34.992496 28660 server_base.cc:1034] running on GCE node
I20260927 15:20:34.993054 28660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:34.993716 28660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:34.994920 28660 hybrid_clock.cc:648] HybridClock initialized: now 1790522434994904 us; error 32 us; skew 500 ppm
Sep 27 15:20:34 dist-test-slave-m4lt krb5kdc[28285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:34.997735 28660 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:20:34.998624 28660 webserver.cc:460] Webserver started at http://127.9.242.131:43377/ using document root <none> and password file <none>
I20260927 15:20:34.999140 28660 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:34.999198 28660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:34.999372 28660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:35.001413 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "195c568931414404926e36b31e64bdc9"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "e02845de54a424f2aca5020ef8c60f5f"
server_key_iv: "0d145d7d72d4a4284e9d604a7bb251fa"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.002013 28660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "195c568931414404926e36b31e64bdc9"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "e02845de54a424f2aca5020ef8c60f5f"
server_key_iv: "0d145d7d72d4a4284e9d604a7bb251fa"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.006049 28660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 15:20:35.008698 28675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.010174 28660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:35.010365 28660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "195c568931414404926e36b31e64bdc9"
format_stamp: "Formatted at 2026-09-27 15:20:34 on dist-test-slave-m4lt"
server_key: "e02845de54a424f2aca5020ef8c60f5f"
server_key_iv: "0d145d7d72d4a4284e9d604a7bb251fa"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.010489 28660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:35.032625 28660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:35.035808 28660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:35.035996 28660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:35.036696 28660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:35.037761 28660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:35.037822 28660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.037881 28660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:35.037912 28660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.046895 28660 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:45867
I20260927 15:20:35.046921 28788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:45867 every 8 connection(s)
I20260927 15:20:35.047819 28660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:35.056383 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 28660
I20260927 15:20:35.056528 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:35.056828 10186 external_mini_cluster.cc:1428] Setting key ca026ff47e8e0ed8868f2824d2ec2575
Sep 27 15:20:35 dist-test-slave-m4lt krb5kdc[28285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522434, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:35.063148 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.049736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:38879 (local address 127.9.242.190:43733)
0927 15:20:35.049886 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.049891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.050799 (+ 908us) server_negotiation.cc:407] Connection header received
0927 15:20:35.052551 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.052554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.052613 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.052688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.054239 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.054770 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.056359 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.056719 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.056835 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.059968 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.059971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.059974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:35.060001 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.061176 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.061702 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.061705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.061707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.061785 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.062091 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.062094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.062095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.062281 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.062364 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.062864 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.062988 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:20:35.063982 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.050167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:43733 (local address 127.9.242.131:38879)
0927 15:20:35.050647 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.050704 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.052309 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.052892 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.052903 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.053397 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:35.054087 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.054105 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.054892 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.054895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.056213 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.056225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.056941 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.056948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.057236 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.058057 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:35.058079 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:35.059811 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:35.061288 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:35.061294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:35.061309 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:35.061585 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:35.061864 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:35.061867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:35.061869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:35.061999 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:35.062375 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:35.062382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:35.062628 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.062793 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.063116 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260927 15:20:35.065104 28789 heartbeater.cc:347] Connected to a master server at 127.9.242.190:43733
I20260927 15:20:35.065349 28789 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:35.065918 28789 heartbeater.cc:511] Master 127.9.242.190:43733 requested a full tablet report, sending...
I20260927 15:20:35.066947 28330 ts_manager.cc:194] Registered new tserver with Master: 195c568931414404926e36b31e64bdc9 (127.9.242.131:45867)
I20260927 15:20:35.067485 28330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:38879
I20260927 15:20:35.070715 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:20:35.080171 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.072632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.9.242.190:43733)
0927 15:20:35.072754 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.072758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.072950 (+ 192us) server_negotiation.cc:407] Connection header received
0927 15:20:35.073063 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.073066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.073108 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.073169 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.073904 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.074411 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.075256 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.075481 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.075563 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.077085 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.077088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.077090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:35.077111 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.078463 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.079003 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.079006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.079008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.079086 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:35.079398 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:35.079401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:35.079402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:35.079603 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.079715 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.079963 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.080053 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:20:35.082929 28330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:35.085929 28330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:35.100142 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.095047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.9.242.129:32991)
0927 15:20:35.095607 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.095628 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.095651 (+ 23us) server_negotiation.cc:407] Connection header received
0927 15:20:35.095729 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.095734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.095958 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.096110 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:35.096480 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.097249 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.098769 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.099511 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.099606 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.099767 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.099905 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.100003 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":116,"threads_started":1}
I20260927 15:20:35.100138 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.094262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:32991 (local address 127.0.0.1:47024)
0927 15:20:35.095235 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.095285 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.095402 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.096119 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.096125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.096148 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:35.096362 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.096370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.097373 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.097376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.098621 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.098632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.099617 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.099620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.099732 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.099768 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.099868 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.099987 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":69,"spinlock_wait_cycles":6272,"thread_start_us":110,"threads_started":1}
I20260927 15:20:35.100644 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.094220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40919 (local address 127.0.0.1:49606)
0927 15:20:35.095235 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.095282 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.095399 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.096140 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.096142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.096154 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:35.096372 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.096379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.097928 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.097932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.099111 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.099121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.100241 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.100245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.100363 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.100384 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.100443 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.100504 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":78,"spinlock_wait_cycles":3968,"thread_start_us":88,"threads_started":1}
I20260927 15:20:35.100847 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.094493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:45867 (local address 127.0.0.1:44164)
0927 15:20:35.095587 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.095599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.095674 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.096257 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.096260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.096286 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:35.096417 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.096424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.097889 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.097892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.099012 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.099021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.100241 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.100245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.100584 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.100604 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.100676 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.100736 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":290,"thread_start_us":92,"threads_started":1}
I20260927 15:20:35.101048 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.094333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.9.242.130:40919)
0927 15:20:35.095675 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.095686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.095707 (+ 21us) server_negotiation.cc:407] Connection header received
0927 15:20:35.095781 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.095786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.095998 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.096504 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:35.096662 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.097802 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.099227 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.099986 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.100066 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.100179 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.100823 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.100921 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1240,"thread_start_us":103,"threads_started":1}
I20260927 15:20:35.101049 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.094838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.9.242.131:45867)
0927 15:20:35.095168 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.095180 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.095767 (+ 587us) server_negotiation.cc:407] Connection header received
0927 15:20:35.095836 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.095841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.095979 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.096109 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:35.096516 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.097731 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.099122 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.099945 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.100048 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.100179 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.100822 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.100922 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
I20260927 15:20:35.103276 28455 tablet_service.cc:1467] Processing CreateTablet for tablet 1937ac95466249d59290f6cdeda3c5cc (DEFAULT_TABLE table=test-table [id=cf35bbd9fccd4f1283b06c5fa0d88466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:35.104184 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1937ac95466249d59290f6cdeda3c5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:35.104203 28723 tablet_service.cc:1467] Processing CreateTablet for tablet 1937ac95466249d59290f6cdeda3c5cc (DEFAULT_TABLE table=test-table [id=cf35bbd9fccd4f1283b06c5fa0d88466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:35.105247 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1937ac95466249d59290f6cdeda3c5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:35.105602 28589 tablet_service.cc:1467] Processing CreateTablet for tablet 1937ac95466249d59290f6cdeda3c5cc (DEFAULT_TABLE table=test-table [id=cf35bbd9fccd4f1283b06c5fa0d88466]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:35.106585 28589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1937ac95466249d59290f6cdeda3c5cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:35.111378 28806 tablet_bootstrap.cc:492] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: Bootstrap starting.
I20260927 15:20:35.111933 28807 tablet_bootstrap.cc:492] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Bootstrap starting.
I20260927 15:20:35.113032 28808 tablet_bootstrap.cc:492] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Bootstrap starting.
I20260927 15:20:35.113590 28806 tablet_bootstrap.cc:654] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:35.114166 28807 tablet_bootstrap.cc:654] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:35.114895 28808 tablet_bootstrap.cc:654] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:35.114976 28806 log.cc:824] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:35.115351 28807 log.cc:824] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:35.115995 28808 log.cc:824] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:35.117103 28806 tablet_bootstrap.cc:492] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: No bootstrap required, opened a new log
I20260927 15:20:35.117201 28807 tablet_bootstrap.cc:492] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: No bootstrap required, opened a new log
I20260927 15:20:35.117291 28806 ts_tablet_manager.cc:1397] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 15:20:35.117388 28807 ts_tablet_manager.cc:1397] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:35.117712 28808 tablet_bootstrap.cc:492] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: No bootstrap required, opened a new log
I20260927 15:20:35.117878 28808 ts_tablet_manager.cc:1397] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:20:35.120118 28806 raft_consensus.cc:348] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.120393 28806 raft_consensus.cc:374] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:35.120469 28806 raft_consensus.cc:729] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05030175aeb645398a0362c4a0012a3a, State: Initialized, Role: FOLLOWER
I20260927 15:20:35.120909 28806 consensus_queue.cc:260] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.121058 28807 raft_consensus.cc:348] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.121299 28807 raft_consensus.cc:374] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:35.121362 28807 raft_consensus.cc:729] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195c568931414404926e36b31e64bdc9, State: Initialized, Role: FOLLOWER
I20260927 15:20:35.121306 28808 raft_consensus.cc:348] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.121527 28808 raft_consensus.cc:374] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:35.121580 28808 raft_consensus.cc:729] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39399cb378b04d83a0557793d8fa2679, State: Initialized, Role: FOLLOWER
I20260927 15:20:35.121882 28806 ts_tablet_manager.cc:1428] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 15:20:35.121832 28807 consensus_queue.cc:260] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.121960 28521 heartbeater.cc:503] Master 127.9.242.190:43733 was elected leader, sending a full tablet report...
I20260927 15:20:35.121999 28808 consensus_queue.cc:260] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.122493 28789 heartbeater.cc:503] Master 127.9.242.190:43733 was elected leader, sending a full tablet report...
I20260927 15:20:35.122743 28808 ts_tablet_manager.cc:1428] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 15:20:35.122763 28522 tablet.cc:2369] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:35.123077 28655 heartbeater.cc:503] Master 127.9.242.190:43733 was elected leader, sending a full tablet report...
I20260927 15:20:35.123353 28807 ts_tablet_manager.cc:1428] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 15:20:35.298794 28790 tablet.cc:2369] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 15:20:35.331470 28656 tablet.cc:2369] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:35.414992 28812 raft_consensus.cc:482] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:35.415221 28812 raft_consensus.cc:504] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.416463 28812 leader_election.cc:288] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39399cb378b04d83a0557793d8fa2679 (127.9.242.130:40919), 195c568931414404926e36b31e64bdc9 (127.9.242.131:45867)
I20260927 15:20:35.421739 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.416736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:41099 (local address 127.9.242.130:40919)
0927 15:20:35.416879 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.416884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.417091 (+ 207us) server_negotiation.cc:407] Connection header received
0927 15:20:35.417294 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.417298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.417361 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.417449 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:35.417926 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.418718 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.420457 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.421065 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.421196 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.421249 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.421501 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.421585 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 15:20:35.421739 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.416792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:52131 (local address 127.9.242.131:45867)
0927 15:20:35.416924 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.416928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.417111 (+ 183us) server_negotiation.cc:407] Connection header received
0927 15:20:35.417327 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.417330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.417380 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.417448 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:35.417922 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.418722 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.420457 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.421086 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.421196 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.421249 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.421510 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.421585 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:20:35.421782 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.416660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40919 (local address 127.9.242.129:41099)
0927 15:20:35.416990 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.417004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.417144 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.417484 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.417488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.417516 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:35.417798 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.417805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.418857 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.418859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.420255 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.420267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.421199 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.421203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.421347 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.421377 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.421458 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.421556 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":178,"spinlock_wait_cycles":21376,"thread_start_us":92,"threads_started":1}
I20260927 15:20:35.421785 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.416716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:45867 (local address 127.9.242.129:52131)
0927 15:20:35.417032 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.417043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.417144 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.417484 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.417488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.417516 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:35.417790 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.417799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.418853 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.418857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.420255 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.420267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.421199 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.421203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.421394 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.421413 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.421473 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.421560 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":187,"spinlock_wait_cycles":21888,"thread_start_us":107,"threads_started":1}
I20260927 15:20:35.422381 28609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937ac95466249d59290f6cdeda3c5cc" candidate_uuid: "05030175aeb645398a0362c4a0012a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39399cb378b04d83a0557793d8fa2679" is_pre_election: true
I20260927 15:20:35.422544 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937ac95466249d59290f6cdeda3c5cc" candidate_uuid: "05030175aeb645398a0362c4a0012a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195c568931414404926e36b31e64bdc9" is_pre_election: true
I20260927 15:20:35.422659 28609 raft_consensus.cc:2454] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05030175aeb645398a0362c4a0012a3a in term 0.
I20260927 15:20:35.422822 28743 raft_consensus.cc:2454] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05030175aeb645398a0362c4a0012a3a in term 0.
I20260927 15:20:35.423113 28409 leader_election.cc:302] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05030175aeb645398a0362c4a0012a3a, 39399cb378b04d83a0557793d8fa2679; no voters:
I20260927 15:20:35.423372 28812 raft_consensus.cc:2781] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:35.423449 28812 raft_consensus.cc:482] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:35.423478 28812 raft_consensus.cc:3037] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:35.424544 28812 raft_consensus.cc:504] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.424875 28812 leader_election.cc:288] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [CANDIDATE]: Term 1 election: Requested vote from peers 39399cb378b04d83a0557793d8fa2679 (127.9.242.130:40919), 195c568931414404926e36b31e64bdc9 (127.9.242.131:45867)
I20260927 15:20:35.425160 28609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937ac95466249d59290f6cdeda3c5cc" candidate_uuid: "05030175aeb645398a0362c4a0012a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39399cb378b04d83a0557793d8fa2679"
I20260927 15:20:35.425221 28743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937ac95466249d59290f6cdeda3c5cc" candidate_uuid: "05030175aeb645398a0362c4a0012a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195c568931414404926e36b31e64bdc9"
I20260927 15:20:35.425272 28609 raft_consensus.cc:3037] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:35.425336 28743 raft_consensus.cc:3037] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:35.426340 28743 raft_consensus.cc:2454] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05030175aeb645398a0362c4a0012a3a in term 1.
I20260927 15:20:35.426376 28609 raft_consensus.cc:2454] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05030175aeb645398a0362c4a0012a3a in term 1.
I20260927 15:20:35.426621 28408 leader_election.cc:302] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05030175aeb645398a0362c4a0012a3a, 195c568931414404926e36b31e64bdc9; no voters:
I20260927 15:20:35.426780 28812 raft_consensus.cc:2781] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:35.427008 28812 raft_consensus.cc:686] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 1 LEADER]: Becoming Leader. State: Replica: 05030175aeb645398a0362c4a0012a3a, State: Running, Role: LEADER
I20260927 15:20:35.427340 28812 consensus_queue.cc:237] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } }
I20260927 15:20:35.430286 28328 catalog_manager.cc:5495] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 05030175aeb645398a0362c4a0012a3a (127.9.242.129). New cstate: current_term: 1 leader_uuid: "05030175aeb645398a0362c4a0012a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05030175aeb645398a0362c4a0012a3a" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 } health_report { overall_health: UNKNOWN } } }
I20260927 15:20:35.464669 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.461376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.9.242.129:32991)
0927 15:20:35.461513 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.461517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.461529 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:20:35.461602 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.461603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.461679 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.461798 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:35.462013 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.462509 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.463366 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.463584 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.463664 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.463917 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:35.464312 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:35.464418 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.464522 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.464558 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 15:20:35.470495 28609 raft_consensus.cc:1261] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 1 FOLLOWER]: Refusing update from remote peer 05030175aeb645398a0362c4a0012a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:35.470495 28743 raft_consensus.cc:1261] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 1 FOLLOWER]: Refusing update from remote peer 05030175aeb645398a0362c4a0012a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:35.471217 28812 consensus_queue.cc:1035] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39399cb378b04d83a0557793d8fa2679" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 40919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:35.471398 28817 consensus_queue.cc:1035] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "195c568931414404926e36b31e64bdc9" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 45867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:35.479189 28822 mvcc.cc:205] Tried to move back new op lower bound from 7333979895679868928 to 7333979895512649728. Current Snapshot: MvccSnapshot[applied={T|T < 7333979895679868928}]
I20260927 15:20:35.488310 28330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:35.488498 28330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:35.490914 28330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ed5ccb40b9e4976a134010b4a0b90fc: Sending DeleteTablet for 3 replicas of tablet 1937ac95466249d59290f6cdeda3c5cc
I20260927 15:20:35.491699 28589 tablet_service.cc:1514] Processing DeleteTablet for tablet 1937ac95466249d59290f6cdeda3c5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:35 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:49606
I20260927 15:20:35.491748 28723 tablet_service.cc:1514] Processing DeleteTablet for tablet 1937ac95466249d59290f6cdeda3c5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:35 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:44164
I20260927 15:20:35.491828 28455 tablet_service.cc:1514] Processing DeleteTablet for tablet 1937ac95466249d59290f6cdeda3c5cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:35 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:47024
I20260927 15:20:35.492266 28831 tablet_replica.cc:330] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: stopping tablet replica
I20260927 15:20:35.492420 28832 tablet_replica.cc:330] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a: stopping tablet replica
I20260927 15:20:35.492666 28831 raft_consensus.cc:2229] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:35.492662 28832 raft_consensus.cc:2229] T 1937ac95466249d59290f6cdeda3c5cc P 05030175aeb645398a0362c4a0012a3a [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:35.492799 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 28392
I20260927 15:20:35.492975 28831 raft_consensus.cc:2258] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:35.494508 28830 tablet_replica.cc:330] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: stopping tablet replica
I20260927 15:20:35.494777 28830 raft_consensus.cc:2229] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:35.495041 28830 raft_consensus.cc:2258] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:35.496402 28830 ts_tablet_manager.cc:1905] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:35.496450 28831 ts_tablet_manager.cc:1905] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:35.498832 28830 ts_tablet_manager.cc:1918] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 15:20:35.498832 28317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:32991 (error 108)
I20260927 15:20:35.498926 28830 log.cc:1196] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/1937ac95466249d59290f6cdeda3c5cc
I20260927 15:20:35.499142 28830 ts_tablet_manager.cc:1939] T 1937ac95466249d59290f6cdeda3c5cc P 39399cb378b04d83a0557793d8fa2679: Deleting consensus metadata
I20260927 15:20:35.499404 28831 ts_tablet_manager.cc:1918] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:35.499559 28831 log.cc:1196] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/1937ac95466249d59290f6cdeda3c5cc
I20260927 15:20:35.499886 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.499597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:32991 (local address 127.0.0.1:47046)
0927 15:20:35.499727 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.499797 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:32991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:20:35.499987 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 28526
I20260927 15:20:35.500011 28831 ts_tablet_manager.cc:1939] T 1937ac95466249d59290f6cdeda3c5cc P 195c568931414404926e36b31e64bdc9: Deleting consensus metadata
I20260927 15:20:35.501133 28314 catalog_manager.cc:4841] TS 195c568931414404926e36b31e64bdc9 (127.9.242.131:45867): tablet 1937ac95466249d59290f6cdeda3c5cc (table test-table [id=cf35bbd9fccd4f1283b06c5fa0d88466]) successfully deleted
W20260927 15:20:35.502697 28317 catalog_manager.cc:4584] TS 05030175aeb645398a0362c4a0012a3a (127.9.242.129:32991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1937ac95466249d59290f6cdeda3c5cc: Network error: Client connection negotiation failed: client connection to 127.9.242.129:32991: connect: Connection refused (error 111)
I20260927 15:20:35.505633 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 28660
I20260927 15:20:35.506558 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.505823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:40919 (local address 127.0.0.1:49612)
0927 15:20:35.506417 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.506473 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.130:40919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518}
I20260927 15:20:35.510502 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 28298
2026-09-27T15:20:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2440 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T15:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:35Z Disabled control of system clock
I20260927 15:20:35.537817 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:44953
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:44953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:20:35.643376 28837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:35.643633 28837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:35.643667 28837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:35.647590 28837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:35.647646 28837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:35.647660 28837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:35.647675 28837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:35.647689 28837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:35.652446 28837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:44953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:35.653486 28837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:35.654891 28837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:35.662914 28842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:35.663067 28837 server_base.cc:1034] running on GCE node
W20260927 15:20:35.662876 28843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:35.662876 28845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:35.663730 28837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:35.664817 28837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:35.665977 28837 hybrid_clock.cc:648] HybridClock initialized: now 1790522435665962 us; error 26 us; skew 500 ppm
I20260927 15:20:35.667775 28837 webserver.cc:460] Webserver started at http://127.9.242.190:41897/ using document root <none> and password file <none>
I20260927 15:20:35.668284 28837 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:35.668340 28837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:35.668497 28837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:35.670497 28837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "01b6b00d76a440f8b8634383e77fbb4a"
format_stamp: "Formatted at 2026-09-27 15:20:35 on dist-test-slave-m4lt"
server_key: "4f4799332786e99660631b64858ef89a"
server_key_iv: "108a495acea4241f79929e6eb7d3afae"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.671003 28837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "01b6b00d76a440f8b8634383e77fbb4a"
format_stamp: "Formatted at 2026-09-27 15:20:35 on dist-test-slave-m4lt"
server_key: "4f4799332786e99660631b64858ef89a"
server_key_iv: "108a495acea4241f79929e6eb7d3afae"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.674451 28837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:35.676744 28851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.677987 28837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:35.678081 28837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "01b6b00d76a440f8b8634383e77fbb4a"
format_stamp: "Formatted at 2026-09-27 15:20:35 on dist-test-slave-m4lt"
server_key: "4f4799332786e99660631b64858ef89a"
server_key_iv: "108a495acea4241f79929e6eb7d3afae"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.678167 28837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:35.702709 28837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:35.703423 28837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:35.703578 28837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:35.710143 28837 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:44953
I20260927 15:20:35.710201 28903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:44953 every 8 connection(s)
I20260927 15:20:35.711118 28837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:35.713079 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 28837
I20260927 15:20:35.713192 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:35.713503 10186 external_mini_cluster.cc:1428] Setting key 656db3190dacc3bc4a49314eafa4d2b0
I20260927 15:20:35.714615 28904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:35.720346 28904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a: Bootstrap starting.
I20260927 15:20:35.722652 28904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:35.722723 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.715047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.9.242.190:44953)
0927 15:20:35.715728 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.715743 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.715779 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:20:35.716494 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.716511 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.716899 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.717200 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.717619 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.718629 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.719498 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.719966 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.720108 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.720985 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.720990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.721003 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:35.721402 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.721461 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:35.721482 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.721564 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.721726 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.721992 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":146,"threads_started":1}
I20260927 15:20:35.723423 28904 log.cc:824] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:35.725448 28904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a: No bootstrap required, opened a new log
I20260927 15:20:35.729097 28904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6b00d76a440f8b8634383e77fbb4a" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44953 } }
I20260927 15:20:35.729336 28904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:35.729393 28904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b6b00d76a440f8b8634383e77fbb4a, State: Initialized, Role: FOLLOWER
I20260927 15:20:35.729823 28904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6b00d76a440f8b8634383e77fbb4a" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44953 } }
I20260927 15:20:35.729956 28904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:35.730020 28904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:35.730122 28904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:35.731019 28904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6b00d76a440f8b8634383e77fbb4a" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44953 } }
I20260927 15:20:35.731354 28904 leader_election.cc:302] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b6b00d76a440f8b8634383e77fbb4a; no voters:
I20260927 15:20:35.731643 28904 leader_election.cc:288] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:35.731760 28909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:35.731967 28909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [term 1 LEADER]: Becoming Leader. State: Replica: 01b6b00d76a440f8b8634383e77fbb4a, State: Running, Role: LEADER
I20260927 15:20:35.732263 28909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6b00d76a440f8b8634383e77fbb4a" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44953 } }
I20260927 15:20:35.732784 28904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:35.733830 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b6b00d76a440f8b8634383e77fbb4a. Latest consensus state: current_term: 1 leader_uuid: "01b6b00d76a440f8b8634383e77fbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6b00d76a440f8b8634383e77fbb4a" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44953 } } }
I20260927 15:20:35.733978 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:35.733827 28910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b6b00d76a440f8b8634383e77fbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6b00d76a440f8b8634383e77fbb4a" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44953 } } }
I20260927 15:20:35.734091 28910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:35.734488 28918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:35.737541 28918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:35.742724 28918 catalog_manager.cc:1347] Generated new cluster ID: 81ce63ce5bfa4361ad1883a8326bc93f
I20260927 15:20:35.742784 28918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:35.761168 28918 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:35.762830 28918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:35.772748 28918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a: Generated new TSK 0
I20260927 15:20:35.773445 28918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:35.786324 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:44953
--builtin_ntp_servers=127.9.242.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:20:35.894243 28928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:35.894500 28928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:35.894537 28928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:35.898502 28928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:35.898554 28928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:35.898615 28928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:20:35.903254 28928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:35.904263 28928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:35.905572 28928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:35.914631 28933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:35.914635 28936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:35.914633 28934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:35.914942 28928 server_base.cc:1034] running on GCE node
I20260927 15:20:35.915334 28928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:35.915946 28928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:35.917111 28928 hybrid_clock.cc:648] HybridClock initialized: now 1790522435917079 us; error 42 us; skew 500 ppm
I20260927 15:20:35.918895 28928 webserver.cc:460] Webserver started at http://127.9.242.129:46305/ using document root <none> and password file <none>
I20260927 15:20:35.919430 28928 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:35.919492 28928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:35.919665 28928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:35.921622 28928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "85933e6b9f354575b46ed57c615ee185"
format_stamp: "Formatted at 2026-09-27 15:20:35 on dist-test-slave-m4lt"
server_key: "ad82577038ec66d61e65d58526d68296"
server_key_iv: "c558c716fdfef6b7aedbc4df27444fa1"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.922178 28928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "85933e6b9f354575b46ed57c615ee185"
format_stamp: "Formatted at 2026-09-27 15:20:35 on dist-test-slave-m4lt"
server_key: "ad82577038ec66d61e65d58526d68296"
server_key_iv: "c558c716fdfef6b7aedbc4df27444fa1"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.925700 28928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:20:35.927968 28942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.929169 28928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:35.929333 28928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "85933e6b9f354575b46ed57c615ee185"
format_stamp: "Formatted at 2026-09-27 15:20:35 on dist-test-slave-m4lt"
server_key: "ad82577038ec66d61e65d58526d68296"
server_key_iv: "c558c716fdfef6b7aedbc4df27444fa1"
server_key_version: "encryptionkey@0"
I20260927 15:20:35.929430 28928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:35.947299 28928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:35.947973 28928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:35.948136 28928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:35.948698 28928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:35.949592 28928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:35.949647 28928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.949735 28928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:35.949759 28928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:35.957998 28928 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:42643
I20260927 15:20:35.958046 29055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:42643 every 8 connection(s)
I20260927 15:20:35.958892 28928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:35.962397 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 28928
I20260927 15:20:35.962507 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:35.962776 10186 external_mini_cluster.cc:1428] Setting key 87a87d5a12c64cfc344fffaf0cfca8bc
I20260927 15:20:35.964672 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:44953
--builtin_ntp_servers=127.9.242.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:20:35.968375 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.960566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:49193 (local address 127.9.242.190:44953)
0927 15:20:35.960719 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:35.960724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:35.961805 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 15:20:35.963170 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:35.963173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:35.963234 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:35.963306 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:35.964571 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.965304 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.966507 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.966743 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.966832 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.967902 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:35.967905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:35.967907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:35.967918 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:35.967937 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:35.967945 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:35.968002 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:35.968143 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:20:35.968218 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:20:35.969228 29058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:35.960957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:44953 (local address 127.9.242.129:49193)
0927 15:20:35.961608 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:35.961677 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:35.962978 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:35.963659 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:35.963667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:35.963786 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:35.964425 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.964441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.965447 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.965450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.966386 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:35.966396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:35.966914 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:35.966919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:35.967059 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:35.967615 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:35.967617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:35.967655 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:35.967716 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:35.968012 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:35.968021 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:35.968027 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:20:35.968135 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 15:20:35.968403 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260927 15:20:35.970352 29056 heartbeater.cc:347] Connected to a master server at 127.9.242.190:44953
I20260927 15:20:35.970595 29056 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:35.971205 29056 heartbeater.cc:511] Master 127.9.242.190:44953 requested a full tablet report, sending...
I20260927 15:20:35.972856 28868 ts_manager.cc:194] Registered new tserver with Master: 85933e6b9f354575b46ed57c615ee185 (127.9.242.129:42643)
I20260927 15:20:35.973985 28868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.129:49193
W20260927 15:20:36.074548 29059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:36.074815 29059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:36.074856 29059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:36.078842 29059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:36.078891 29059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:36.078949 29059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:20:36.083789 29059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:36.084830 29059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:36.086165 29059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:36.095494 29064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.095505 29065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.095522 29067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:36.095830 29059 server_base.cc:1034] running on GCE node
I20260927 15:20:36.096289 29059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:36.096977 29059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:36.098129 29059 hybrid_clock.cc:648] HybridClock initialized: now 1790522436098107 us; error 32 us; skew 500 ppm
I20260927 15:20:36.099963 29059 webserver.cc:460] Webserver started at http://127.9.242.130:41395/ using document root <none> and password file <none>
I20260927 15:20:36.100533 29059 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:36.100590 29059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:36.100782 29059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:36.102839 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "677119e82b694696aea4d8472a2b4b0c"
format_stamp: "Formatted at 2026-09-27 15:20:36 on dist-test-slave-m4lt"
server_key: "f4c16f0f6010daf11e5506c6bb9c5909"
server_key_iv: "c3f1b7eb44c1fa4db9ecc1eaac77d1d3"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.103377 29059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "677119e82b694696aea4d8472a2b4b0c"
format_stamp: "Formatted at 2026-09-27 15:20:36 on dist-test-slave-m4lt"
server_key: "f4c16f0f6010daf11e5506c6bb9c5909"
server_key_iv: "c3f1b7eb44c1fa4db9ecc1eaac77d1d3"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.106928 29059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:36.109251 29073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.110496 29059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:36.110606 29059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "677119e82b694696aea4d8472a2b4b0c"
format_stamp: "Formatted at 2026-09-27 15:20:36 on dist-test-slave-m4lt"
server_key: "f4c16f0f6010daf11e5506c6bb9c5909"
server_key_iv: "c3f1b7eb44c1fa4db9ecc1eaac77d1d3"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.110699 29059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:36.139541 29059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:36.140229 29059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:36.140403 29059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:36.140992 29059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:36.141984 29059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:36.142031 29059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.142083 29059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:36.142112 29059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.150434 29059 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:41751
I20260927 15:20:36.150455 29186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:41751 every 8 connection(s)
I20260927 15:20:36.151297 29059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:36.160041 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.153070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:60327 (local address 127.9.242.190:44953)
0927 15:20:36.153214 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.153218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.154269 (+ 1051us) server_negotiation.cc:407] Connection header received
0927 15:20:36.155619 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.155623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.155680 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.155751 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:36.156620 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.157149 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.158301 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.158514 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.158600 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.159583 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:36.159586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:36.159588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:36.159600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:36.159617 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:36.159624 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:36.159681 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.159812 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.159876 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:20:36.160910 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.153479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:44953 (local address 127.9.242.130:60327)
0927 15:20:36.154126 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.154167 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.155383 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.155925 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.155932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.156015 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:36.156488 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.156501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.157259 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.157262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.158187 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.158197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.158644 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.158648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.158799 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.159311 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:36.159312 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:36.159352 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:36.159445 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:36.159689 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:36.159698 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:36.159704 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:20:36.159805 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 15:20:36.160073 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":128,"threads_started":1}
I20260927 15:20:36.160941 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 29059
I20260927 15:20:36.161046 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:36.161360 10186 external_mini_cluster.cc:1428] Setting key deeb45254a3af0db347f2cec91b67323
I20260927 15:20:36.162055 29187 heartbeater.cc:347] Connected to a master server at 127.9.242.190:44953
I20260927 15:20:36.162271 29187 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:36.162837 29187 heartbeater.cc:511] Master 127.9.242.190:44953 requested a full tablet report, sending...
I20260927 15:20:36.163380 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:44953
--builtin_ntp_servers=127.9.242.148:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:20:36.163863 28868 ts_manager.cc:194] Registered new tserver with Master: 677119e82b694696aea4d8472a2b4b0c (127.9.242.130:41751)
I20260927 15:20:36.164434 28868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.130:60327
W20260927 15:20:36.269345 29190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:36.269584 29190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:36.269621 29190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:36.273623 29190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:36.273689 29190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:36.273752 29190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:20:36.278393 29190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:36.279369 29190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:36.280709 29190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:36.289852 29198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.289865 29196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:36.289836 29195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:36.289902 29190 server_base.cc:1034] running on GCE node
I20260927 15:20:36.290403 29190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:36.290970 29190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:36.292141 29190 hybrid_clock.cc:648] HybridClock initialized: now 1790522436292123 us; error 29 us; skew 500 ppm
I20260927 15:20:36.293915 29190 webserver.cc:460] Webserver started at http://127.9.242.131:35371/ using document root <none> and password file <none>
I20260927 15:20:36.294441 29190 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:36.294503 29190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:36.294680 29190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:36.296622 29190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "d120009fc39544409c8140eb73f9e7c3"
format_stamp: "Formatted at 2026-09-27 15:20:36 on dist-test-slave-m4lt"
server_key: "a04a993d67cbe09e8870433a183a1167"
server_key_iv: "c1de6f1fc20a8d519917b1cb4b07faa0"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.297142 29190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "d120009fc39544409c8140eb73f9e7c3"
format_stamp: "Formatted at 2026-09-27 15:20:36 on dist-test-slave-m4lt"
server_key: "a04a993d67cbe09e8870433a183a1167"
server_key_iv: "c1de6f1fc20a8d519917b1cb4b07faa0"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.300640 29190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260927 15:20:36.303018 29204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.304224 29190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:20:36.304333 29190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "d120009fc39544409c8140eb73f9e7c3"
format_stamp: "Formatted at 2026-09-27 15:20:36 on dist-test-slave-m4lt"
server_key: "a04a993d67cbe09e8870433a183a1167"
server_key_iv: "c1de6f1fc20a8d519917b1cb4b07faa0"
server_key_version: "encryptionkey@0"
I20260927 15:20:36.304428 29190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:36.325014 29190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:36.325753 29190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:36.325938 29190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:36.326510 29190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:36.327451 29190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:36.327497 29190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.327548 29190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:36.327574 29190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:36.336215 29190 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:33689
I20260927 15:20:36.336216 29317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:33689 every 8 connection(s)
I20260927 15:20:36.337127 29190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:36.339181 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 29190
I20260927 15:20:36.339296 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:36.339584 10186 external_mini_cluster.cc:1428] Setting key 8a60b3174de1cab4a25a691032103b4d
I20260927 15:20:36.346382 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.338936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:54051 (local address 127.9.242.190:44953)
0927 15:20:36.339061 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.339064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.340426 (+ 1362us) server_negotiation.cc:407] Connection header received
0927 15:20:36.341971 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.341974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.342030 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.342097 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:36.342976 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.343516 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.344626 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.344847 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.344931 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.345924 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:36.345928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:36.345930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:36.345944 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:36.345963 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:36.345970 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:36.346026 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.346163 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.346237 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:20:36.347255 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.339387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:44953 (local address 127.9.242.131:54051)
0927 15:20:36.340221 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.340275 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.341776 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.342262 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.342269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.342364 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:36.342842 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.342854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.343640 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.343642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.344512 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.344521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.344982 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.344986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.345140 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.345683 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:36.345684 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:36.345724 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:36.345787 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:36.346036 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:36.346044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:36.346050 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:20:36.346154 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:20:36.346521 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":138,"threads_started":1}
I20260927 15:20:36.348445 29318 heartbeater.cc:347] Connected to a master server at 127.9.242.190:44953
I20260927 15:20:36.348649 29318 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:36.349125 29318 heartbeater.cc:511] Master 127.9.242.190:44953 requested a full tablet report, sending...
I20260927 15:20:36.350096 28868 ts_manager.cc:194] Registered new tserver with Master: d120009fc39544409c8140eb73f9e7c3 (127.9.242.131:33689)
I20260927 15:20:36.350622 28868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.131:54051
I20260927 15:20:36.353526 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:20:36.358891 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.355436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.9.242.190:44953)
0927 15:20:36.355570 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.355573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.355644 (+ 71us) server_negotiation.cc:407] Connection header received
0927 15:20:36.355738 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.355740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.355777 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.355838 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:36.356054 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.356572 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.357356 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.357594 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.357694 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.358455 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:36.358458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:36.358460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:36.358471 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:36.358489 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:36.358497 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:36.358591 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.358718 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.358750 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:20:36.361537 28868 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:36.364470 28868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:36.378225 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.372701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.9.242.130:41751)
0927 15:20:36.373533 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.373550 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.373574 (+ 24us) server_negotiation.cc:407] Connection header received
0927 15:20:36.373648 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.373676 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.373906 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.374334 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:36.374463 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.375354 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.376859 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.377553 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.377648 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.377876 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.377978 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.378071 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":89,"threads_started":1}
I20260927 15:20:36.378221 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.372576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:41751 (local address 127.0.0.1:36076)
0927 15:20:36.372969 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.373013 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.373130 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.374074 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.374080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.374102 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:36.374215 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.374220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.375447 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.375450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.376745 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.376755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.377685 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.377688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.377802 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.377850 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 15:20:36.377952 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 15:20:36.378070 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":22,"thread_start_us":79,"threads_started":1}
I20260927 15:20:36.378381 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.372655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:42643 (local address 127.0.0.1:55590)
0927 15:20:36.373756 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.373770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.373896 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.374394 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.374397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.374412 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:36.374494 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.374500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.375847 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.375851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.376982 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.376988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.377927 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.377931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.378019 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.378046 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 15:20:36.378095 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 15:20:36.378165 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":62,"thread_start_us":96,"threads_started":1}
I20260927 15:20:36.378897 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.373086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:33689 (local address 127.0.0.1:38746)
0927 15:20:36.374082 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.374094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.374165 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.374790 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.374793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.374811 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:36.374887 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.374892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.376116 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.376119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.377191 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.377201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.378183 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.378186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.378266 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.378658 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 15:20:36.378707 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 15:20:36.378773 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":84,"threads_started":1}
I20260927 15:20:36.379606 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.373375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.9.242.131:33689)
0927 15:20:36.374400 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.374410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.374432 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:20:36.374499 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.374504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.374668 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.374801 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:36.374995 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.376001 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.377313 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.378072 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.379164 (+ 1092us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.379332 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.379377 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.379461 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":114,"threads_started":1}
I20260927 15:20:36.380442 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.373387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.9.242.129:42643)
0927 15:20:36.373799 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.373815 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.373861 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:20:36.374018 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.374023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.374252 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.374410 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:36.374581 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.375693 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.377081 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.377788 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.378603 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.378791 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.378839 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.378927 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":115,"threads_started":1}
I20260927 15:20:36.381377 29121 tablet_service.cc:1467] Processing CreateTablet for tablet 693482f773f542cf95b0c088b06de2e3 (DEFAULT_TABLE table=test-table [id=fd3b446ee2c2426cae2f3fe8ff566030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:36.382221 29252 tablet_service.cc:1467] Processing CreateTablet for tablet 693482f773f542cf95b0c088b06de2e3 (DEFAULT_TABLE table=test-table [id=fd3b446ee2c2426cae2f3fe8ff566030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:36.382567 29121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693482f773f542cf95b0c088b06de2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:36.383046 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693482f773f542cf95b0c088b06de2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:36.383078 28990 tablet_service.cc:1467] Processing CreateTablet for tablet 693482f773f542cf95b0c088b06de2e3 (DEFAULT_TABLE table=test-table [id=fd3b446ee2c2426cae2f3fe8ff566030]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:36.384037 28990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693482f773f542cf95b0c088b06de2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:36.388484 29335 tablet_bootstrap.cc:492] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Bootstrap starting.
I20260927 15:20:36.389989 29336 tablet_bootstrap.cc:492] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Bootstrap starting.
I20260927 15:20:36.390935 29335 tablet_bootstrap.cc:654] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:36.391110 29337 tablet_bootstrap.cc:492] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: Bootstrap starting.
I20260927 15:20:36.392099 29336 tablet_bootstrap.cc:654] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:36.392148 29335 log.cc:824] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:36.392854 29336 log.cc:824] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:36.392874 29337 tablet_bootstrap.cc:654] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:36.393595 29337 log.cc:824] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:36.394291 29335 tablet_bootstrap.cc:492] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: No bootstrap required, opened a new log
I20260927 15:20:36.394486 29335 ts_tablet_manager.cc:1397] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 15:20:36.394942 29336 tablet_bootstrap.cc:492] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: No bootstrap required, opened a new log
I20260927 15:20:36.395124 29336 ts_tablet_manager.cc:1397] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:20:36.398689 29335 raft_consensus.cc:348] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.398959 29335 raft_consensus.cc:374] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:36.398986 29337 tablet_bootstrap.cc:492] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: No bootstrap required, opened a new log
I20260927 15:20:36.399024 29335 raft_consensus.cc:729] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d120009fc39544409c8140eb73f9e7c3, State: Initialized, Role: FOLLOWER
I20260927 15:20:36.399150 29337 ts_tablet_manager.cc:1397] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260927 15:20:36.399053 29336 raft_consensus.cc:348] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.399284 29336 raft_consensus.cc:374] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:36.399336 29336 raft_consensus.cc:729] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677119e82b694696aea4d8472a2b4b0c, State: Initialized, Role: FOLLOWER
I20260927 15:20:36.399526 29335 consensus_queue.cc:260] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.399801 29336 consensus_queue.cc:260] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.400519 29335 ts_tablet_manager.cc:1428] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:20:36.400597 29336 ts_tablet_manager.cc:1428] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:36.400893 29318 heartbeater.cc:503] Master 127.9.242.190:44953 was elected leader, sending a full tablet report...
I20260927 15:20:36.401142 29187 heartbeater.cc:503] Master 127.9.242.190:44953 was elected leader, sending a full tablet report...
W20260927 15:20:36.402132 29188 tablet.cc:2369] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:36.403184 29337 raft_consensus.cc:348] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.403474 29337 raft_consensus.cc:374] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:36.403530 29337 raft_consensus.cc:729] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85933e6b9f354575b46ed57c615ee185, State: Initialized, Role: FOLLOWER
I20260927 15:20:36.403990 29337 consensus_queue.cc:260] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.404584 29056 heartbeater.cc:503] Master 127.9.242.190:44953 was elected leader, sending a full tablet report...
I20260927 15:20:36.404848 29337 ts_tablet_manager.cc:1428] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 15:20:36.460038 29057 tablet.cc:2369] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:36.502993 29341 raft_consensus.cc:482] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:36.503196 29341 raft_consensus.cc:504] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.504323 29341 leader_election.cc:288] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 677119e82b694696aea4d8472a2b4b0c (127.9.242.130:41751), 85933e6b9f354575b46ed57c615ee185 (127.9.242.129:42643)
I20260927 15:20:36.507886 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.504547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:41751 (local address 127.9.242.131:37465)
0927 15:20:36.504679 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.504695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.504786 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.505123 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.505127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.505144 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:36.505245 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.505250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.506006 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.506009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.506850 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.506858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.507492 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.507495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.507587 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.507610 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:20:36.507676 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:20:36.507738 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":39}
I20260927 15:20:36.507937 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.504787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:37465 (local address 127.9.242.130:41751)
0927 15:20:36.504923 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.504926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.504938 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:20:36.504983 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.504985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.505023 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.505096 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:36.505352 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.505902 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.506973 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.507385 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.507474 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.507510 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.507700 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.507772 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 15:20:36.508029 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.504661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:49935 (local address 127.9.242.129:42643)
0927 15:20:36.504794 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.504799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.505044 (+ 245us) server_negotiation.cc:407] Connection header received
0927 15:20:36.505127 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.505130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.505173 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.505232 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:36.505479 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.506030 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.507096 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.507512 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.507620 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.507661 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.507844 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.507926 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:20:36.508242 29344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.504547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:42643 (local address 127.9.242.131:49935)
0927 15:20:36.504958 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.504967 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:36.505049 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:36.505248 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:36.505250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:36.505264 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:36.505376 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.505381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.506126 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.506129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.506988 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:36.506998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.507628 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.507631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:36.507719 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.507764 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 15:20:36.507830 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 15:20:36.507892 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":117,"thread_start_us":112,"threads_started":1}
I20260927 15:20:36.508618 29141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693482f773f542cf95b0c088b06de2e3" candidate_uuid: "d120009fc39544409c8140eb73f9e7c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "677119e82b694696aea4d8472a2b4b0c" is_pre_election: true
I20260927 15:20:36.508728 29010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693482f773f542cf95b0c088b06de2e3" candidate_uuid: "d120009fc39544409c8140eb73f9e7c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85933e6b9f354575b46ed57c615ee185" is_pre_election: true
I20260927 15:20:36.508899 29141 raft_consensus.cc:2454] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d120009fc39544409c8140eb73f9e7c3 in term 0.
I20260927 15:20:36.509002 29010 raft_consensus.cc:2454] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d120009fc39544409c8140eb73f9e7c3 in term 0.
I20260927 15:20:36.509361 29207 leader_election.cc:302] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 677119e82b694696aea4d8472a2b4b0c, d120009fc39544409c8140eb73f9e7c3; no voters:
I20260927 15:20:36.509627 29341 raft_consensus.cc:2781] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:36.509737 29341 raft_consensus.cc:482] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:36.509789 29341 raft_consensus.cc:3037] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:36.510748 29341 raft_consensus.cc:504] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.511061 29341 leader_election.cc:288] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [CANDIDATE]: Term 1 election: Requested vote from peers 677119e82b694696aea4d8472a2b4b0c (127.9.242.130:41751), 85933e6b9f354575b46ed57c615ee185 (127.9.242.129:42643)
I20260927 15:20:36.511373 29141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693482f773f542cf95b0c088b06de2e3" candidate_uuid: "d120009fc39544409c8140eb73f9e7c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "677119e82b694696aea4d8472a2b4b0c"
I20260927 15:20:36.511381 29010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693482f773f542cf95b0c088b06de2e3" candidate_uuid: "d120009fc39544409c8140eb73f9e7c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85933e6b9f354575b46ed57c615ee185"
I20260927 15:20:36.511484 29010 raft_consensus.cc:3037] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:36.511488 29141 raft_consensus.cc:3037] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:36.512455 29141 raft_consensus.cc:2454] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d120009fc39544409c8140eb73f9e7c3 in term 1.
I20260927 15:20:36.512563 29010 raft_consensus.cc:2454] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d120009fc39544409c8140eb73f9e7c3 in term 1.
I20260927 15:20:36.512732 29207 leader_election.cc:302] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 677119e82b694696aea4d8472a2b4b0c, d120009fc39544409c8140eb73f9e7c3; no voters:
I20260927 15:20:36.512943 29341 raft_consensus.cc:2781] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:36.513155 29341 raft_consensus.cc:686] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 1 LEADER]: Becoming Leader. State: Replica: d120009fc39544409c8140eb73f9e7c3, State: Running, Role: LEADER
I20260927 15:20:36.513453 29341 consensus_queue.cc:237] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } }
I20260927 15:20:36.516481 28868 catalog_manager.cc:5495] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d120009fc39544409c8140eb73f9e7c3 (127.9.242.131). New cstate: current_term: 1 leader_uuid: "d120009fc39544409c8140eb73f9e7c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d120009fc39544409c8140eb73f9e7c3" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 33689 } health_report { overall_health: HEALTHY } } }
I20260927 15:20:36.535872 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.532078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.9.242.131:33689)
0927 15:20:36.532203 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:36.532207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:36.532220 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:20:36.532395 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:36.532399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:36.532461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:36.532549 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:36.532753 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.533288 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.534335 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:36.534625 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:36.534710 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:36.534961 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:36.535416 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:36.535534 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:36.535670 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 15:20:36.535704 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:20:36.541826 29010 raft_consensus.cc:1261] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 1 FOLLOWER]: Refusing update from remote peer d120009fc39544409c8140eb73f9e7c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:36.541826 29141 raft_consensus.cc:1261] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 1 FOLLOWER]: Refusing update from remote peer d120009fc39544409c8140eb73f9e7c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:36.542451 29345 consensus_queue.cc:1035] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "677119e82b694696aea4d8472a2b4b0c" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 41751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:36.542618 29341 consensus_queue.cc:1035] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85933e6b9f354575b46ed57c615ee185" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 42643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:36.550540 29348 mvcc.cc:205] Tried to move back new op lower bound from 7333979900067778560 to 7333979899961962496. Current Snapshot: MvccSnapshot[applied={T|T < 7333979900067778560}]
I20260927 15:20:36.552968 29349 mvcc.cc:205] Tried to move back new op lower bound from 7333979900067778560 to 7333979899961962496. Current Snapshot: MvccSnapshot[applied={T|T < 7333979900067778560 or (T in {7333979900067778560})}]
I20260927 15:20:36.559460 28867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:36.559638 28867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:36.562049 28867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01b6b00d76a440f8b8634383e77fbb4a: Sending DeleteTablet for 3 replicas of tablet 693482f773f542cf95b0c088b06de2e3
I20260927 15:20:36.562777 29121 tablet_service.cc:1514] Processing DeleteTablet for tablet 693482f773f542cf95b0c088b06de2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:36 UTC) from {username='slave'} at 127.0.0.1:36076
I20260927 15:20:36.562860 29252 tablet_service.cc:1514] Processing DeleteTablet for tablet 693482f773f542cf95b0c088b06de2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:36 UTC) from {username='slave'} at 127.0.0.1:38746
I20260927 15:20:36.562862 28990 tablet_service.cc:1514] Processing DeleteTablet for tablet 693482f773f542cf95b0c088b06de2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:36 UTC) from {username='slave'} at 127.0.0.1:55590
I20260927 15:20:36.563330 29358 tablet_replica.cc:330] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: stopping tablet replica
I20260927 15:20:36.563403 29360 tablet_replica.cc:330] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185: stopping tablet replica
I20260927 15:20:36.563660 29360 raft_consensus.cc:2229] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:36.563661 29358 raft_consensus.cc:2229] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:36.563994 29358 raft_consensus.cc:2258] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:36.564004 29360 raft_consensus.cc:2258] T 693482f773f542cf95b0c088b06de2e3 P 85933e6b9f354575b46ed57c615ee185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:36.564098 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 28928
I20260927 15:20:36.564339 29359 tablet_replica.cc:330] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: stopping tablet replica
I20260927 15:20:36.564591 29359 raft_consensus.cc:2229] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:36.564917 29359 raft_consensus.cc:2258] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:36.566465 29359 ts_tablet_manager.cc:1905] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:36.567189 29358 ts_tablet_manager.cc:1905] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:36.569229 29359 ts_tablet_manager.cc:1918] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:36.569326 29359 log.cc:1196] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/693482f773f542cf95b0c088b06de2e3
I20260927 15:20:36.569634 29359 ts_tablet_manager.cc:1939] T 693482f773f542cf95b0c088b06de2e3 P d120009fc39544409c8140eb73f9e7c3: Deleting consensus metadata
I20260927 15:20:36.569761 29358 ts_tablet_manager.cc:1918] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:36.569846 29358 log.cc:1196] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/693482f773f542cf95b0c088b06de2e3
I20260927 15:20:36.570104 29358 ts_tablet_manager.cc:1939] T 693482f773f542cf95b0c088b06de2e3 P 677119e82b694696aea4d8472a2b4b0c: Deleting consensus metadata
I20260927 15:20:36.570863 28855 catalog_manager.cc:4841] TS d120009fc39544409c8140eb73f9e7c3 (127.9.242.131:33689): tablet 693482f773f542cf95b0c088b06de2e3 (table test-table [id=fd3b446ee2c2426cae2f3fe8ff566030]) successfully deleted
I20260927 15:20:36.571156 28854 catalog_manager.cc:4841] TS 677119e82b694696aea4d8472a2b4b0c (127.9.242.130:41751): tablet 693482f773f542cf95b0c088b06de2e3 (table test-table [id=fd3b446ee2c2426cae2f3fe8ff566030]) successfully deleted
W20260927 15:20:36.574559 28853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:42643 (error 108)
I20260927 15:20:36.574719 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 29059
I20260927 15:20:36.576532 29330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:36.576211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:42643 (local address 127.0.0.1:55604)
0927 15:20:36.576381 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:36.576445 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:42643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 15:20:36.576691 28853 catalog_manager.cc:4584] TS 85933e6b9f354575b46ed57c615ee185 (127.9.242.129:42643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 693482f773f542cf95b0c088b06de2e3: Network error: Client connection negotiation failed: client connection to 127.9.242.129:42643: connect: Connection refused (error 111)
I20260927 15:20:36.579485 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 29190
I20260927 15:20:36.584440 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 28837
2026-09-27T15:20:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1073 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:36 dist-test-slave-m4lt krb5kdc[29364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:36 dist-test-slave-m4lt krb5kdc[29364](info): set up 2 sockets
Sep 27 15:20:36 dist-test-slave-m4lt krb5kdc[29364](info): commencing operation
krb5kdc: starting...
W20260927 15:20:37.541476 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:37 dist-test-slave-m4lt krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:20:37.656523 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36327
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:36327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:37.763984 29377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:37.764262 29377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:37.764321 29377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:37.768363 29377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:37.768424 29377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:37.768451 29377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:37.768472 29377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:37.768498 29377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:37.773778 29377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:36327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:37.774771 29377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:37.775820 29377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:37.784358 29385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:37.784363 29383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:37.784402 29382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:37.784695 29377 server_base.cc:1034] running on GCE node
I20260927 15:20:37.785246 29377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:37.786393 29377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:37.787549 29377 hybrid_clock.cc:648] HybridClock initialized: now 1790522437787527 us; error 35 us; skew 500 ppm
Sep 27 15:20:37 dist-test-slave-m4lt krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522437, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:37.790207 29377 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:20:37.791085 29377 webserver.cc:460] Webserver started at http://127.9.242.190:46663/ using document root <none> and password file <none>
I20260927 15:20:37.791607 29377 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:37.791671 29377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:37.791869 29377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:37.793915 29377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "f41f3c808c694092bf1e389dd8fae9a3"
format_stamp: "Formatted at 2026-09-27 15:20:37 on dist-test-slave-m4lt"
server_key: "8f3ebfb5b634e972185fb8e5aa2a7a15"
server_key_iv: "ccd9553b4f1e30944b0e407ad4851a16"
server_key_version: "encryptionkey@0"
I20260927 15:20:37.794426 29377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "f41f3c808c694092bf1e389dd8fae9a3"
format_stamp: "Formatted at 2026-09-27 15:20:37 on dist-test-slave-m4lt"
server_key: "8f3ebfb5b634e972185fb8e5aa2a7a15"
server_key_iv: "ccd9553b4f1e30944b0e407ad4851a16"
server_key_version: "encryptionkey@0"
I20260927 15:20:37.797999 29377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:37.800360 29392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:37.801791 29377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:37.801924 29377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "f41f3c808c694092bf1e389dd8fae9a3"
format_stamp: "Formatted at 2026-09-27 15:20:37 on dist-test-slave-m4lt"
server_key: "8f3ebfb5b634e972185fb8e5aa2a7a15"
server_key_iv: "ccd9553b4f1e30944b0e407ad4851a16"
server_key_version: "encryptionkey@0"
I20260927 15:20:37.802026 29377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:37.820982 29377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:37.824296 29377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:37.824505 29377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:37.831627 29377 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:36327
I20260927 15:20:37.831655 29444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:36327 every 8 connection(s)
I20260927 15:20:37.832899 29377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:37.835762 29445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:37.841161 29445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3: Bootstrap starting.
I20260927 15:20:37.842494 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 29377
I20260927 15:20:37.842607 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:37.842877 10186 external_mini_cluster.cc:1428] Setting key a514959f9c1ec358327592cf8000503f
I20260927 15:20:37.843775 29445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:37.844492 29445 log.cc:824] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:37.846736 29445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3: No bootstrap required, opened a new log
I20260927 15:20:37.850796 29445 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41f3c808c694092bf1e389dd8fae9a3" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36327 } }
I20260927 15:20:37.851078 29445 raft_consensus.cc:374] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:37.851150 29445 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41f3c808c694092bf1e389dd8fae9a3, State: Initialized, Role: FOLLOWER
I20260927 15:20:37.851662 29445 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41f3c808c694092bf1e389dd8fae9a3" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36327 } }
Sep 27 15:20:37 dist-test-slave-m4lt krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:37.851821 29445 raft_consensus.cc:388] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:37.851897 29445 raft_consensus.cc:482] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:37.852005 29445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:37.852949 29445 raft_consensus.cc:504] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41f3c808c694092bf1e389dd8fae9a3" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36327 } }
I20260927 15:20:37.853286 29445 leader_election.cc:302] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41f3c808c694092bf1e389dd8fae9a3; no voters:
I20260927 15:20:37.853590 29445 leader_election.cc:288] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:37.853709 29450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:37.853915 29450 raft_consensus.cc:686] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [term 1 LEADER]: Becoming Leader. State: Replica: f41f3c808c694092bf1e389dd8fae9a3, State: Running, Role: LEADER
I20260927 15:20:37.854225 29450 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41f3c808c694092bf1e389dd8fae9a3" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36327 } }
I20260927 15:20:37.854931 29445 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:37.857684 29451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f41f3c808c694092bf1e389dd8fae9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41f3c808c694092bf1e389dd8fae9a3" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36327 } } }
I20260927 15:20:37.857829 29452 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41f3c808c694092bf1e389dd8fae9a3. Latest consensus state: current_term: 1 leader_uuid: "f41f3c808c694092bf1e389dd8fae9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41f3c808c694092bf1e389dd8fae9a3" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36327 } } }
I20260927 15:20:37.857896 29451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:37.857928 29452 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:37.857987 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:37.843984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.9.242.190:36327)
0927 15:20:37.844450 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:37.844466 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:37.844500 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:20:37.845279 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:37.845307 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:37.845622 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:37.845964 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:37.846820 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:37.847848 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:37.849008 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:37.849477 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:37.849584 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:37.852421 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:37.852426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:37.852440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:37.852466 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:37.854404 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:37.854884 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:37.854890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:37.854895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:37.855031 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:37.855433 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:37.855436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:37.855438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:37.855904 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:37.856072 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:37.856442 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 15:20:37.856750 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":131,"threads_started":1}
W20260927 15:20:37.860423 29465 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:37.860502 29465 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:37.860566 29466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:37.861274 29466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:37.866739 29466 catalog_manager.cc:1347] Generated new cluster ID: 77f45e8d48f4420786405c76bdff8c99
I20260927 15:20:37.866801 29466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:37.896812 29466 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:37.898187 29466 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:37.903074 29466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3: Generated new TSK 0
I20260927 15:20:37.903812 29466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.242.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
Entry for principal kudu/127.9.242.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab.
I20260927 15:20:37.935037 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36327
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:38.044028 29471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:38.044298 29471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:38.044359 29471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:38.048350 29471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:38.048416 29471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:38.048496 29471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:20:38.053743 29471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.129.keytab
--principal=kudu/127.9.242.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:38.054813 29471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:38.055853 29471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:38.065303 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:38.065312 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:38.065536 29471 server_base.cc:1034] running on GCE node
W20260927 15:20:38.065668 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:38.066032 29471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:38.066648 29471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:38.067811 29471 hybrid_clock.cc:648] HybridClock initialized: now 1790522438067782 us; error 39 us; skew 500 ppm
Sep 27 15:20:38 dist-test-slave-m4lt krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:38.070516 29471 init.cc:377] Logged in from keytab as kudu/127.9.242.129@KRBTEST.COM (short username kudu)
I20260927 15:20:38.071398 29471 webserver.cc:460] Webserver started at http://127.9.242.129:33233/ using document root <none> and password file <none>
I20260927 15:20:38.071890 29471 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:38.071944 29471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:38.072144 29471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:38.074321 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "d4224509c60a431eb76617d0d3198c0f"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "65b1c66e053f855e2f17b4e6bb7889df"
server_key_iv: "8979e86acf5acc0b1b90c680d4f38fad"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.074872 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "d4224509c60a431eb76617d0d3198c0f"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "65b1c66e053f855e2f17b4e6bb7889df"
server_key_iv: "8979e86acf5acc0b1b90c680d4f38fad"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.078698 29471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:20:38.081193 29486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.082525 29471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:38.082617 29471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "d4224509c60a431eb76617d0d3198c0f"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "65b1c66e053f855e2f17b4e6bb7889df"
server_key_iv: "8979e86acf5acc0b1b90c680d4f38fad"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.082700 29471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:38.094014 29471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:38.097719 29471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:38.097911 29471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:38.098592 29471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:38.099459 29471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:38.099505 29471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.099556 29471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:38.099584 29471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.108141 29471 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:35171
I20260927 15:20:38.108187 29599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:35171 every 8 connection(s)
I20260927 15:20:38.108982 29471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:38.111528 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 29471
I20260927 15:20:38.111653 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:38.111989 10186 external_mini_cluster.cc:1428] Setting key 4f9bec442f15af74053d9ecc9152a3f5
Sep 27 15:20:38 dist-test-slave-m4lt krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.129@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.242.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.130@KRBTEST.COM" created.
I20260927 15:20:38.123127 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.110710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:60259 (local address 127.9.242.190:36327)
0927 15:20:38.110830 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.110834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.111743 (+ 909us) server_negotiation.cc:407] Connection header received
0927 15:20:38.113423 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.113426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.113477 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.113546 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.114781 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.115321 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.116578 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.116942 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.117044 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.119737 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:38.119742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:38.119745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:38.119775 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:38.121156 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.121666 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.121669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.121671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.121751 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.122062 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.122064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.122065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.122261 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:38.122333 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.122818 (+ 485us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.122956 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:20:38.123997 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.111018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36327 (local address 127.9.242.129:60259)
0927 15:20:38.111561 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.111623 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.113114 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.113761 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.113768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.114103 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.114595 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.114609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.115434 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.115438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.116459 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.116468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.117090 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.117093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.117324 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.117917 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.117933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.119561 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:38.121258 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:38.121263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:38.121276 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:38.121572 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:38.121859 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:38.121862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:38.121863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:38.121990 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:38.122359 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:38.122365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:38.122596 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.122766 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.123084 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":101,"threads_started":1}
I20260927 15:20:38.125219 29600 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36327
I20260927 15:20:38.125501 29600 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:38.126000 29600 heartbeater.cc:511] Master 127.9.242.190:36327 requested a full tablet report, sending...
I20260927 15:20:38.127380 29409 ts_manager.cc:194] Registered new tserver with Master: d4224509c60a431eb76617d0d3198c0f (127.9.242.129:35171)
I20260927 15:20:38.128445 29409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.129@KRBTEST.COM'} at 127.9.242.129:60259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
Entry for principal kudu/127.9.242.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab.
I20260927 15:20:38.136708 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36327
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:38.243445 29605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:38.243710 29605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:38.243753 29605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:38.247680 29605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:38.247735 29605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:38.247795 29605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:20:38.252952 29605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.130.keytab
--principal=kudu/127.9.242.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:38.254015 29605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:38.255199 29605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:38.264947 29610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:38.264961 29613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:38.264961 29611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:38.265189 29605 server_base.cc:1034] running on GCE node
I20260927 15:20:38.265571 29605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:38.266220 29605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:38.267377 29605 hybrid_clock.cc:648] HybridClock initialized: now 1790522438267358 us; error 33 us; skew 500 ppm
Sep 27 15:20:38 dist-test-slave-m4lt krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:38.269979 29605 init.cc:377] Logged in from keytab as kudu/127.9.242.130@KRBTEST.COM (short username kudu)
I20260927 15:20:38.270848 29605 webserver.cc:460] Webserver started at http://127.9.242.130:45897/ using document root <none> and password file <none>
I20260927 15:20:38.271400 29605 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:38.271461 29605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:38.271621 29605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:38.273612 29605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "420805ad65ff47e9b1be66d0d76a0a27"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "6e483586f164c83b53955180231ce330"
server_key_iv: "97802b9dca5021c2ff1b88c624fdca92"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.274138 29605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "420805ad65ff47e9b1be66d0d76a0a27"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "6e483586f164c83b53955180231ce330"
server_key_iv: "97802b9dca5021c2ff1b88c624fdca92"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.277608 29605 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:38.279958 29620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.281212 29605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:38.281368 29605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "420805ad65ff47e9b1be66d0d76a0a27"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "6e483586f164c83b53955180231ce330"
server_key_iv: "97802b9dca5021c2ff1b88c624fdca92"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.281455 29605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:38.292701 29605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:38.295390 29605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:38.295526 29605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:38.296022 29605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:38.296756 29605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:38.296799 29605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.296836 29605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:38.296855 29605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.305049 29605 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:43365
I20260927 15:20:38.305079 29733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:43365 every 8 connection(s)
I20260927 15:20:38.305956 29605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:38.312320 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 29605
I20260927 15:20:38.312448 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:38.312708 10186 external_mini_cluster.cc:1428] Setting key 44621facdb4ee21179bf7baa0936c91a
Sep 27 15:20:38 dist-test-slave-m4lt krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.130@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:38.319211 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.307663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:33273 (local address 127.9.242.190:36327)
0927 15:20:38.307869 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.307874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.308605 (+ 731us) server_negotiation.cc:407] Connection header received
0927 15:20:38.309910 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.309912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.309957 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.310027 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.311109 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.311630 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.313000 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.313219 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.313304 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.315816 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:38.315819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:38.315821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:38.315844 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:38.317287 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.317797 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.317800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.317802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.317883 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.318178 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.318181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.318182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.318362 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:38.318441 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.318927 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.319045 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:20:38.319957 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.308045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36327 (local address 127.9.242.130:33273)
0927 15:20:38.308470 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.308510 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.309722 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.310177 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.310183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.310546 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.311003 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.311015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.311723 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.311725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.312890 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.312900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.313401 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.313406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.313551 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.314108 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.314126 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.315647 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:38.317409 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:38.317415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:38.317430 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:38.317706 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:38.317960 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:38.317963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:38.317965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:38.318091 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:38.318455 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:38.318462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:38.318696 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.318857 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.319162 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260927 15:20:38.321113 29734 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36327
WARNING: no policy specified for kudu/127.9.242.131@KRBTEST.COM; defaulting to no policy
I20260927 15:20:38.321362 29734 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:38.321954 29734 heartbeater.cc:511] Master 127.9.242.190:36327 requested a full tablet report, sending...
I20260927 15:20:38.322968 29409 ts_manager.cc:194] Registered new tserver with Master: 420805ad65ff47e9b1be66d0d76a0a27 (127.9.242.130:43365)
I20260927 15:20:38.323613 29409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.130@KRBTEST.COM'} at 127.9.242.130:33273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
Entry for principal kudu/127.9.242.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab.
I20260927 15:20:38.338459 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36327
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:38.447026 29739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:38.447278 29739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:38.447319 29739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:38.451297 29739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:38.451356 29739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:38.451419 29739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:20:38.456693 29739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.131.keytab
--principal=kudu/127.9.242.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:36327
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:38.457707 29739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:38.458673 29739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:38.467872 29744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:38.467943 29739 server_base.cc:1034] running on GCE node
W20260927 15:20:38.467890 29745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:38.467903 29747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:38.468461 29739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:38.469066 29739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:38.470227 29739 hybrid_clock.cc:648] HybridClock initialized: now 1790522438470209 us; error 27 us; skew 500 ppm
Sep 27 15:20:38 dist-test-slave-m4lt krb5kdc[29364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:38.472842 29739 init.cc:377] Logged in from keytab as kudu/127.9.242.131@KRBTEST.COM (short username kudu)
I20260927 15:20:38.473630 29739 webserver.cc:460] Webserver started at http://127.9.242.131:40565/ using document root <none> and password file <none>
I20260927 15:20:38.474154 29739 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:38.474217 29739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:38.474401 29739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:38.476363 29739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "8e1986fc916f4d2299c3d82abbfa8c81"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "a0ce90a657f0d7df5fe18ca988b3bc1f"
server_key_iv: "ab9b08cd94d0370f68a8784fa16c691b"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.476871 29739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "8e1986fc916f4d2299c3d82abbfa8c81"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "a0ce90a657f0d7df5fe18ca988b3bc1f"
server_key_iv: "ab9b08cd94d0370f68a8784fa16c691b"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.480456 29739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:20:38.482859 29754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.484074 29739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:20:38.484175 29739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "8e1986fc916f4d2299c3d82abbfa8c81"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "a0ce90a657f0d7df5fe18ca988b3bc1f"
server_key_iv: "ab9b08cd94d0370f68a8784fa16c691b"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.484319 29739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:38.511251 29739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:38.514097 29739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:38.514282 29739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:38.514847 29739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:38.515638 29739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:38.515686 29739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.515736 29739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:38.515763 29739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.523900 29739 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:46801
I20260927 15:20:38.523931 29867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:46801 every 8 connection(s)
I20260927 15:20:38.524729 29739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
Sep 27 15:20:38 dist-test-slave-m4lt krb5kdc[29364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522438, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.131@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:38.534431 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 29739
I20260927 15:20:38.534543 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:38.534781 10186 external_mini_cluster.cc:1428] Setting key 8ae4ba8c7ddafdf575cba683a2999635
I20260927 15:20:38.538147 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.526577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:47647 (local address 127.9.242.190:36327)
0927 15:20:38.526703 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.526707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.527486 (+ 779us) server_negotiation.cc:407] Connection header received
0927 15:20:38.528698 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.528701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.528756 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.528830 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.530017 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.530570 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.531780 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.532006 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.532095 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.534590 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:38.534594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:38.534597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:38.534627 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:38.536203 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.536679 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.536682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.536684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.536776 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.537118 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.537120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.537122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.537313 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:38.537385 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.537931 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.538032 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:20:38.538905 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.526899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:36327 (local address 127.9.242.131:47647)
0927 15:20:38.527330 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.527373 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.528507 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.528984 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.528991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.529352 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:38.529883 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.529898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.530700 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.530703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.531630 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.531640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.532171 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.532175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.532339 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.532917 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:20:38.532941 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:38.534420 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:38.536313 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:38.536318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:38.536330 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:38.536590 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:38.536868 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:20:38.536871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:20:38.536872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:20:38.537045 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:20:38.537428 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:38.537433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:38.537692 (+ 259us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.537864 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.538117 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260927 15:20:38.539947 29868 heartbeater.cc:347] Connected to a master server at 127.9.242.190:36327
I20260927 15:20:38.540163 29868 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:38.540612 29868 heartbeater.cc:511] Master 127.9.242.190:36327 requested a full tablet report, sending...
I20260927 15:20:38.541539 29409 ts_manager.cc:194] Registered new tserver with Master: 8e1986fc916f4d2299c3d82abbfa8c81 (127.9.242.131:46801)
I20260927 15:20:38.542122 29409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.242.131@KRBTEST.COM'} at 127.9.242.131:47647
I20260927 15:20:38.548424 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:20:38.557171 29448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.550295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.9.242.190:36327)
0927 15:20:38.550425 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.550430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.550596 (+ 166us) server_negotiation.cc:407] Connection header received
0927 15:20:38.550704 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.550706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.550752 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.550813 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.551455 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.551967 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.552747 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.552967 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.553049 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.554313 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:38.554317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:38.554319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:38.554340 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:38.555689 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.556050 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.556054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.556056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.556132 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:38.556366 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:38.556370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:38.556372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:38.556583 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:38.556718 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.556921 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.557022 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:20:38.559983 29409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:38.563071 29409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:38.577032 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.571535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:46801 (local address 127.0.0.1:49548)
0927 15:20:38.572367 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.572378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.572460 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.573142 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.573148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.573182 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:38.573354 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.573359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.574343 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.574347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.575422 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.575432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.576498 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.576501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.576611 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.576647 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.576750 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.576882 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":27,"spinlock_wait_cycles":5760,"thread_start_us":73,"threads_started":1}
I20260927 15:20:38.577241 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.572016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.9.242.131:46801)
0927 15:20:38.572724 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.572739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.572758 (+ 19us) server_negotiation.cc:407] Connection header received
0927 15:20:38.572821 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.572826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.572979 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.573142 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:38.573462 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.574226 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.575557 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.576398 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.576486 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.576634 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.576789 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.576890 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":90,"threads_started":1}
I20260927 15:20:38.577610 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.571224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35171 (local address 127.0.0.1:39496)
0927 15:20:38.572268 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.572321 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.572462 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.573214 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.573216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.573235 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:38.573339 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.573346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.574904 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.574908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.576182 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.576192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.577202 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.577205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.577335 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.577356 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.577404 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.577470 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":47,"spinlock_wait_cycles":896,"thread_start_us":85,"threads_started":1}
I20260927 15:20:38.577893 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.571324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:43365 (local address 127.0.0.1:39764)
0927 15:20:38.572269 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.572321 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.572462 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.573157 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.573160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.573185 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:38.573391 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.573399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.574846 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.574849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.575998 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.576007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.577191 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.577195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.577484 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.577509 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.577582 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.577756 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":269,"spinlock_wait_cycles":896,"thread_start_us":99,"threads_started":1}
I20260927 15:20:38.577896 29882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.571561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.9.242.130:43365)
0927 15:20:38.571993 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.572005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.572408 (+ 403us) server_negotiation.cc:407] Connection header received
0927 15:20:38.572626 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.572631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.572851 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.572973 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:38.573494 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.574694 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.576107 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.576891 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.576990 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.577121 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.577642 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.577765 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":102,"threads_started":1}
I20260927 15:20:38.578310 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.572000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.9.242.129:35171)
0927 15:20:38.572752 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.572764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.572786 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:20:38.572857 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.572862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.573075 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.573495 (+ 420us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:38.573638 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.574777 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.576311 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.577086 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.577904 (+ 818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.578046 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.578093 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.578177 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":135,"threads_started":1}
I20260927 15:20:38.580478 29534 tablet_service.cc:1467] Processing CreateTablet for tablet e16f7501063645959c565ff15c3d0877 (DEFAULT_TABLE table=test-table [id=35f5331b4d5146c0a3f0ddba4b9bfea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:38.580413 29668 tablet_service.cc:1467] Processing CreateTablet for tablet e16f7501063645959c565ff15c3d0877 (DEFAULT_TABLE table=test-table [id=35f5331b4d5146c0a3f0ddba4b9bfea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:38.580413 29802 tablet_service.cc:1467] Processing CreateTablet for tablet e16f7501063645959c565ff15c3d0877 (DEFAULT_TABLE table=test-table [id=35f5331b4d5146c0a3f0ddba4b9bfea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:38.581435 29534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16f7501063645959c565ff15c3d0877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:38.581458 29802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16f7501063645959c565ff15c3d0877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:38.581458 29668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e16f7501063645959c565ff15c3d0877. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:38.587890 29885 tablet_bootstrap.cc:492] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f: Bootstrap starting.
I20260927 15:20:38.589184 29887 tablet_bootstrap.cc:492] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Bootstrap starting.
I20260927 15:20:38.589989 29886 tablet_bootstrap.cc:492] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Bootstrap starting.
I20260927 15:20:38.590152 29885 tablet_bootstrap.cc:654] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:38.591168 29885 log.cc:824] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:38.591189 29887 tablet_bootstrap.cc:654] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:38.591954 29886 tablet_bootstrap.cc:654] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:38.592406 29887 log.cc:824] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:38.593071 29885 tablet_bootstrap.cc:492] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f: No bootstrap required, opened a new log
I20260927 15:20:38.593164 29886 log.cc:824] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:38.593264 29885 ts_tablet_manager.cc:1397] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:38.594491 29887 tablet_bootstrap.cc:492] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: No bootstrap required, opened a new log
I20260927 15:20:38.594671 29887 ts_tablet_manager.cc:1397] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:38.595114 29886 tablet_bootstrap.cc:492] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: No bootstrap required, opened a new log
I20260927 15:20:38.595290 29886 ts_tablet_manager.cc:1397] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 15:20:38.596480 29885 raft_consensus.cc:348] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.596698 29885 raft_consensus.cc:374] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:38.596767 29885 raft_consensus.cc:729] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4224509c60a431eb76617d0d3198c0f, State: Initialized, Role: FOLLOWER
I20260927 15:20:38.597203 29885 consensus_queue.cc:260] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.598102 29885 ts_tablet_manager.cc:1428] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:38.598444 29600 heartbeater.cc:503] Master 127.9.242.190:36327 was elected leader, sending a full tablet report...
I20260927 15:20:38.598472 29887 raft_consensus.cc:348] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.598723 29887 raft_consensus.cc:374] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:38.598825 29887 raft_consensus.cc:729] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1986fc916f4d2299c3d82abbfa8c81, State: Initialized, Role: FOLLOWER
I20260927 15:20:38.598948 29886 raft_consensus.cc:348] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.599167 29886 raft_consensus.cc:374] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:38.599216 29886 raft_consensus.cc:729] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420805ad65ff47e9b1be66d0d76a0a27, State: Initialized, Role: FOLLOWER
I20260927 15:20:38.599238 29887 consensus_queue.cc:260] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.599624 29886 consensus_queue.cc:260] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.600219 29887 ts_tablet_manager.cc:1428] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:38.600502 29886 ts_tablet_manager.cc:1428] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:38.600633 29868 heartbeater.cc:503] Master 127.9.242.190:36327 was elected leader, sending a full tablet report...
I20260927 15:20:38.600852 29734 heartbeater.cc:503] Master 127.9.242.190:36327 was elected leader, sending a full tablet report...
W20260927 15:20:38.610003 29601 tablet.cc:2369] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:38.672883 29893 raft_consensus.cc:482] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:38.673111 29893 raft_consensus.cc:504] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.674247 29893 leader_election.cc:288] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4224509c60a431eb76617d0d3198c0f (127.9.242.129:35171), 8e1986fc916f4d2299c3d82abbfa8c81 (127.9.242.131:46801)
I20260927 15:20:38.678081 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.674621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:51483 (local address 127.9.242.131:46801)
0927 15:20:38.674760 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.674764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.675051 (+ 287us) server_negotiation.cc:407] Connection header received
0927 15:20:38.675148 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.675150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.675201 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.675281 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:38.675574 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.676089 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.677100 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.677513 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.677621 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.677720 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.677902 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.677972 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 15:20:38.678081 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.674531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35171 (local address 127.9.242.130:45971)
0927 15:20:38.674715 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.674729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.674819 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.675060 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.675063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.675080 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:38.675288 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.675296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.676090 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.676093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.676909 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.676918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.677591 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.677595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.677743 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.677786 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.677841 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.677910 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":4}
I20260927 15:20:38.678081 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.674701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:45971 (local address 127.9.242.129:35171)
0927 15:20:38.674816 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.674820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.674835 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:20:38.674910 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.674913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.674964 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.675052 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:38.675423 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.675991 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.677044 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.677476 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.677573 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.677636 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.677883 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.677956 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:20:38.678093 29894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.674531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:46801 (local address 127.9.242.130:51483)
0927 15:20:38.674950 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.674963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:38.675061 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:38.675322 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:38.675325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:38.675341 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:38.675474 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.675480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.676211 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.676214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.676973 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:38.676984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.677627 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.677630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:38.677792 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.677809 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 15:20:38.677879 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 15:20:38.677947 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":49,"thread_start_us":156,"threads_started":1}
I20260927 15:20:38.678778 29822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f7501063645959c565ff15c3d0877" candidate_uuid: "420805ad65ff47e9b1be66d0d76a0a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" is_pre_election: true
I20260927 15:20:38.678762 29554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f7501063645959c565ff15c3d0877" candidate_uuid: "420805ad65ff47e9b1be66d0d76a0a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4224509c60a431eb76617d0d3198c0f" is_pre_election: true
I20260927 15:20:38.679067 29554 raft_consensus.cc:2454] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420805ad65ff47e9b1be66d0d76a0a27 in term 0.
I20260927 15:20:38.679067 29822 raft_consensus.cc:2454] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420805ad65ff47e9b1be66d0d76a0a27 in term 0.
I20260927 15:20:38.679567 29623 leader_election.cc:302] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 420805ad65ff47e9b1be66d0d76a0a27, 8e1986fc916f4d2299c3d82abbfa8c81, d4224509c60a431eb76617d0d3198c0f; no voters:
I20260927 15:20:38.679857 29893 raft_consensus.cc:2781] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:38.679929 29893 raft_consensus.cc:482] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:38.679971 29893 raft_consensus.cc:3037] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:38.680948 29893 raft_consensus.cc:504] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.681252 29893 leader_election.cc:288] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [CANDIDATE]: Term 1 election: Requested vote from peers d4224509c60a431eb76617d0d3198c0f (127.9.242.129:35171), 8e1986fc916f4d2299c3d82abbfa8c81 (127.9.242.131:46801)
I20260927 15:20:38.681604 29822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f7501063645959c565ff15c3d0877" candidate_uuid: "420805ad65ff47e9b1be66d0d76a0a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1986fc916f4d2299c3d82abbfa8c81"
I20260927 15:20:38.681744 29822 raft_consensus.cc:3037] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:38.682878 29822 raft_consensus.cc:2454] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420805ad65ff47e9b1be66d0d76a0a27 in term 1.
I20260927 15:20:38.683152 29623 leader_election.cc:302] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 420805ad65ff47e9b1be66d0d76a0a27, 8e1986fc916f4d2299c3d82abbfa8c81; no voters:
I20260927 15:20:38.683147 29554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e16f7501063645959c565ff15c3d0877" candidate_uuid: "420805ad65ff47e9b1be66d0d76a0a27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4224509c60a431eb76617d0d3198c0f"
I20260927 15:20:38.683280 29554 raft_consensus.cc:3037] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:38.683324 29893 raft_consensus.cc:2781] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:38.683569 29893 raft_consensus.cc:686] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 1 LEADER]: Becoming Leader. State: Replica: 420805ad65ff47e9b1be66d0d76a0a27, State: Running, Role: LEADER
I20260927 15:20:38.683933 29893 consensus_queue.cc:237] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } }
I20260927 15:20:38.684351 29554 raft_consensus.cc:2454] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420805ad65ff47e9b1be66d0d76a0a27 in term 1.
I20260927 15:20:38.687130 29405 catalog_manager.cc:5495] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 420805ad65ff47e9b1be66d0d76a0a27 (127.9.242.130). New cstate: current_term: 1 leader_uuid: "420805ad65ff47e9b1be66d0d76a0a27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "420805ad65ff47e9b1be66d0d76a0a27" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 43365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20260927 15:20:38.697847 29882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.694564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.9.242.130:43365)
0927 15:20:38.694661 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.694665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.694675 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:20:38.694789 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.694791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.694848 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.694932 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:38.695112 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.695629 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.696502 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.696708 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.696780 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.697107 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:38.697500 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:38.697585 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.697632 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.697705 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:20:38.703119 29554 raft_consensus.cc:1261] T e16f7501063645959c565ff15c3d0877 P d4224509c60a431eb76617d0d3198c0f [term 1 FOLLOWER]: Refusing update from remote peer 420805ad65ff47e9b1be66d0d76a0a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:38.703119 29822 raft_consensus.cc:1261] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 1 FOLLOWER]: Refusing update from remote peer 420805ad65ff47e9b1be66d0d76a0a27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:38.703754 29893 consensus_queue.cc:1035] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4224509c60a431eb76617d0d3198c0f" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 35171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:38.703938 29895 consensus_queue.cc:1035] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1986fc916f4d2299c3d82abbfa8c81" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:38.711597 29900 mvcc.cc:205] Tried to move back new op lower bound from 7333979908921614336 to 7333979908851974144. Current Snapshot: MvccSnapshot[applied={T|T < 7333979908921614336}]
I20260927 15:20:38.720487 29408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:38.720649 29408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:38.722934 29408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f41f3c808c694092bf1e389dd8fae9a3: Sending DeleteTablet for 3 replicas of tablet e16f7501063645959c565ff15c3d0877
I20260927 15:20:38.723690 29668 tablet_service.cc:1514] Processing DeleteTablet for tablet e16f7501063645959c565ff15c3d0877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:38 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:39764
I20260927 15:20:38.723690 29534 tablet_service.cc:1514] Processing DeleteTablet for tablet e16f7501063645959c565ff15c3d0877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:38 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:39496
I20260927 15:20:38.724195 29909 tablet_replica.cc:330] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: stopping tablet replica
I20260927 15:20:38.724361 29802 tablet_service.cc:1514] Processing DeleteTablet for tablet e16f7501063645959c565ff15c3d0877 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:38 UTC) from {username='kudu', principal='kudu/127.9.242.190@KRBTEST.COM'} at 127.0.0.1:49548
I20260927 15:20:38.724637 29909 raft_consensus.cc:2229] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:38.724831 29910 tablet_replica.cc:330] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: stopping tablet replica
I20260927 15:20:38.724859 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 29471
I20260927 15:20:38.725121 29910 raft_consensus.cc:2229] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:38.725481 29910 raft_consensus.cc:2258] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:38.726169 29909 raft_consensus.cc:2258] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:38.727025 29910 ts_tablet_manager.cc:1905] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:38.727650 29909 ts_tablet_manager.cc:1905] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:38.729631 29910 ts_tablet_manager.cc:1918] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:38.729751 29910 log.cc:1196] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/e16f7501063645959c565ff15c3d0877
I20260927 15:20:38.730028 29910 ts_tablet_manager.cc:1939] T e16f7501063645959c565ff15c3d0877 P 8e1986fc916f4d2299c3d82abbfa8c81: Deleting consensus metadata
I20260927 15:20:38.730397 29909 ts_tablet_manager.cc:1918] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:38.730535 29909 log.cc:1196] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/e16f7501063645959c565ff15c3d0877
I20260927 15:20:38.730878 29909 ts_tablet_manager.cc:1939] T e16f7501063645959c565ff15c3d0877 P 420805ad65ff47e9b1be66d0d76a0a27: Deleting consensus metadata
I20260927 15:20:38.731145 29395 catalog_manager.cc:4841] TS 8e1986fc916f4d2299c3d82abbfa8c81 (127.9.242.131:46801): tablet e16f7501063645959c565ff15c3d0877 (table test-table [id=35f5331b4d5146c0a3f0ddba4b9bfea2]) successfully deleted
I20260927 15:20:38.731948 29396 catalog_manager.cc:4841] TS 420805ad65ff47e9b1be66d0d76a0a27 (127.9.242.130:43365): tablet e16f7501063645959c565ff15c3d0877 (table test-table [id=35f5331b4d5146c0a3f0ddba4b9bfea2]) successfully deleted
W20260927 15:20:38.732945 29394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:35171 (error 108)
I20260927 15:20:38.733148 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 29605
I20260927 15:20:38.735014 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.734731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:35171 (local address 127.0.0.1:39504)
0927 15:20:38.734864 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:38.734931 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 15:20:38.735172 29394 catalog_manager.cc:4584] TS d4224509c60a431eb76617d0d3198c0f (127.9.242.129:35171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e16f7501063645959c565ff15c3d0877: Network error: Client connection negotiation failed: client connection to 127.9.242.129:35171: connect: Connection refused (error 111)
I20260927 15:20:38.738155 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 29739
I20260927 15:20:38.742666 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 29377
2026-09-27T15:20:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2158 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T15:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:38Z Disabled control of system clock
I20260927 15:20:38.769706 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37789
--webserver_interface=127.9.242.190
--webserver_port=0
--builtin_ntp_servers=127.9.242.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:37789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:20:38.877810 29915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:38.878082 29915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:38.878118 29915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:38.882107 29915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:38.882166 29915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:38.882180 29915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:38.882195 29915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:38.882210 29915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:38.887073 29915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:37789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:37789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:38.888062 29915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:38.889323 29915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:38.897989 29920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:38.897995 29921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:38.898079 29915 server_base.cc:1034] running on GCE node
W20260927 15:20:38.897984 29923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:38.898828 29915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:38.899847 29915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:38.901017 29915 hybrid_clock.cc:648] HybridClock initialized: now 1790522438901002 us; error 25 us; skew 500 ppm
I20260927 15:20:38.902827 29915 webserver.cc:460] Webserver started at http://127.9.242.190:46285/ using document root <none> and password file <none>
I20260927 15:20:38.903380 29915 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:38.903442 29915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:38.903617 29915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:38.905611 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "bc8469a2ca8345aa8bac82e6ff10b73c"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "f5e2ebdc15952f4df492d4b8691a452d"
server_key_iv: "3735f345b8ef54075632fc2ae1ccde73"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.906132 29915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "bc8469a2ca8345aa8bac82e6ff10b73c"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "f5e2ebdc15952f4df492d4b8691a452d"
server_key_iv: "3735f345b8ef54075632fc2ae1ccde73"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.909572 29915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:38.911938 29929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:38.913136 29915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:38.913234 29915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "bc8469a2ca8345aa8bac82e6ff10b73c"
format_stamp: "Formatted at 2026-09-27 15:20:38 on dist-test-slave-m4lt"
server_key: "f5e2ebdc15952f4df492d4b8691a452d"
server_key_iv: "3735f345b8ef54075632fc2ae1ccde73"
server_key_version: "encryptionkey@0"
I20260927 15:20:38.913342 29915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:38.932792 29915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:38.933682 29915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:38.933900 29915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:38.942229 29915 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:37789
I20260927 15:20:38.942247 29981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:37789 every 8 connection(s)
I20260927 15:20:38.943408 29915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:38.946108 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 29915
I20260927 15:20:38.946221 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:38.946471 10186 external_mini_cluster.cc:1428] Setting key dfc8c1f63fbf0567deb8fe9243306f07
I20260927 15:20:38.946694 29982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:38.952658 29982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c: Bootstrap starting.
I20260927 15:20:38.954952 29982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:38.955325 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:38.947710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.9.242.190:37789)
0927 15:20:38.948360 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:38.948381 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:38.948424 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:20:38.949152 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:38.949171 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:38.949574 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:38.949928 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:38.950335 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.951290 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.952131 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:38.952595 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:38.952733 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:38.953581 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:38.953586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:38.953598 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:38.954033 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:38.954103 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:38.954123 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:38.954204 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:38.954345 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 15:20:38.954590 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
I20260927 15:20:38.955725 29982 log.cc:824] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:38.957779 29982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c: No bootstrap required, opened a new log
I20260927 15:20:38.961061 29982 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37789 } }
I20260927 15:20:38.961279 29982 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:38.961315 29982 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8469a2ca8345aa8bac82e6ff10b73c, State: Initialized, Role: FOLLOWER
I20260927 15:20:38.961737 29982 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37789 } }
I20260927 15:20:38.961903 29982 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:38.961951 29982 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:38.962023 29982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:38.962891 29982 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37789 } }
I20260927 15:20:38.963213 29982 leader_election.cc:302] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc8469a2ca8345aa8bac82e6ff10b73c; no voters:
I20260927 15:20:38.963516 29982 leader_election.cc:288] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:38.963578 29987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:38.963762 29987 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [term 1 LEADER]: Becoming Leader. State: Replica: bc8469a2ca8345aa8bac82e6ff10b73c, State: Running, Role: LEADER
I20260927 15:20:38.964051 29987 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37789 } }
I20260927 15:20:38.964790 29982 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:38.965764 29989 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37789 } } }
I20260927 15:20:38.965891 29989 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:38.966187 29988 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc8469a2ca8345aa8bac82e6ff10b73c. Latest consensus state: current_term: 1 leader_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8469a2ca8345aa8bac82e6ff10b73c" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 37789 } } }
I20260927 15:20:38.966274 29988 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:38.966398 29996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:38.969568 29996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:38.975219 29996 catalog_manager.cc:1347] Generated new cluster ID: feb6a764372849608ef29b0244e613d9
I20260927 15:20:38.975281 29996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:38.991230 29996 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:38.992141 29996 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:39.000597 29996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c: Generated new TSK 0
I20260927 15:20:39.001302 29996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:39.008677 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.129:0
--local_ip_for_outbound_sockets=127.9.242.129
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37789
--builtin_ntp_servers=127.9.242.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 15:20:39.116161 30006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:39.116421 30006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:39.116465 30006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:39.120463 30006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:39.120518 30006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:39.120574 30006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.129
I20260927 15:20:39.125247 30006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.242.129
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.129
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:39.126273 30006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:39.127575 30006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:39.136505 30014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:39.136519 30011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:39.136509 30012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:39.136636 30006 server_base.cc:1034] running on GCE node
I20260927 15:20:39.137100 30006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:39.137712 30006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:39.138862 30006 hybrid_clock.cc:648] HybridClock initialized: now 1790522439138838 us; error 41 us; skew 500 ppm
I20260927 15:20:39.140590 30006 webserver.cc:460] Webserver started at http://127.9.242.129:45265/ using document root <none> and password file <none>
I20260927 15:20:39.141119 30006 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:39.141184 30006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:39.141374 30006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:39.143358 30006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data/instance:
uuid: "607387fd7869437bb51c8f2245f4e21e"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "bf7a1f5691a6e7d034d03dfc3dac39b4"
server_key_iv: "8e7f65c59369c96c8bbbbd49ee1a1957"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.143867 30006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance:
uuid: "607387fd7869437bb51c8f2245f4e21e"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "bf7a1f5691a6e7d034d03dfc3dac39b4"
server_key_iv: "8e7f65c59369c96c8bbbbd49ee1a1957"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.147444 30006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:20:39.149873 30020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.151003 30006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:20:39.151105 30006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/wal
uuid: "607387fd7869437bb51c8f2245f4e21e"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "bf7a1f5691a6e7d034d03dfc3dac39b4"
server_key_iv: "8e7f65c59369c96c8bbbbd49ee1a1957"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.151198 30006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:39.193485 30006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:39.194223 30006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:39.194410 30006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:39.195003 30006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:39.195955 30006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:39.196008 30006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.196062 30006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:39.196089 30006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.204784 30006 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.129:36935
I20260927 15:20:39.204854 30133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.129:36935 every 8 connection(s)
I20260927 15:20:39.205736 30006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/data/info.pb
I20260927 15:20:39.214964 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.207574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.129:46785 (local address 127.9.242.190:37789)
0927 15:20:39.207714 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.207718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.208844 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 15:20:39.210289 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.210292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.210338 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.210411 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.211295 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.211857 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.213017 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.213284 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.213371 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.214437 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:39.214440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:39.214442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:39.214454 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:39.214473 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:39.214481 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:39.214538 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.214701 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.214810 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:20:39.215072 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30006
I20260927 15:20:39.215173 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-0/wal/instance
I20260927 15:20:39.215462 10186 external_mini_cluster.cc:1428] Setting key 9550357cbb8ccdfa1efa17d61786139e
I20260927 15:20:39.215787 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.208050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37789 (local address 127.9.242.129:46785)
0927 15:20:39.208694 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.208735 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.210084 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.210562 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.210570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.210657 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:39.211164 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.211176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.211957 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.211960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.212827 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.212836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.213419 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.213422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.213564 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.214180 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:39.214182 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.214225 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:39.214291 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:39.214551 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:39.214556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:39.214560 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.214660 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.214892 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260927 15:20:39.216995 30134 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37789
I20260927 15:20:39.217218 30134 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:39.217528 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.130:0
--local_ip_for_outbound_sockets=127.9.242.130
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37789
--builtin_ntp_servers=127.9.242.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:20:39.217824 30134 heartbeater.cc:511] Master 127.9.242.190:37789 requested a full tablet report, sending...
I20260927 15:20:39.219383 29946 ts_manager.cc:194] Registered new tserver with Master: 607387fd7869437bb51c8f2245f4e21e (127.9.242.129:36935)
I20260927 15:20:39.220482 29946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.129:46785
W20260927 15:20:39.324838 30137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:39.325109 30137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:39.325158 30137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:39.329304 30137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:39.329372 30137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:39.329447 30137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.130
I20260927 15:20:39.334362 30137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.242.130
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.130
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:39.335446 30137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:39.336843 30137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:39.346606 30143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:39.346637 30142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:39.346648 30137 server_base.cc:1034] running on GCE node
W20260927 15:20:39.346889 30145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:39.347301 30137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:39.347919 30137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:39.349115 30137 hybrid_clock.cc:648] HybridClock initialized: now 1790522439349095 us; error 30 us; skew 500 ppm
I20260927 15:20:39.351028 30137 webserver.cc:460] Webserver started at http://127.9.242.130:36175/ using document root <none> and password file <none>
I20260927 15:20:39.351569 30137 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:39.351629 30137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:39.351830 30137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:39.353876 30137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data/instance:
uuid: "fde29fbc9a8a49ea95c9a53585630a79"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "cd19f79c78eeea28003438094f4182b2"
server_key_iv: "77869bb31805efaba4ae6b98f69bdc33"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.354416 30137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance:
uuid: "fde29fbc9a8a49ea95c9a53585630a79"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "cd19f79c78eeea28003438094f4182b2"
server_key_iv: "77869bb31805efaba4ae6b98f69bdc33"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.357972 30137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:39.360276 30151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.361492 30137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:39.361585 30137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal
uuid: "fde29fbc9a8a49ea95c9a53585630a79"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "cd19f79c78eeea28003438094f4182b2"
server_key_iv: "77869bb31805efaba4ae6b98f69bdc33"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.361714 30137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:39.382874 30137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:39.383561 30137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:39.383726 30137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:39.384279 30137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:39.385172 30137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:39.385221 30137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.385258 30137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:39.385272 30137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.393746 30137 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.130:39223
I20260927 15:20:39.393781 30264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.130:39223 every 8 connection(s)
I20260927 15:20:39.394604 30137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/data/info.pb
I20260927 15:20:39.403396 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.396391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.130:52045 (local address 127.9.242.190:37789)
0927 15:20:39.396525 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.396529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.397686 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 15:20:39.399001 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.399005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.399055 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.399130 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.400005 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.400535 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.401622 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.401909 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.401991 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.402963 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:39.402966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:39.402968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:39.402980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:39.402999 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:39.403008 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:39.403063 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.403205 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.403262 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:20:39.403561 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30137
I20260927 15:20:39.403687 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal/instance
I20260927 15:20:39.404003 10186 external_mini_cluster.cc:1428] Setting key e733ddb652c4c0022a1e1223656ba898
I20260927 15:20:39.404475 30267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.396797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37789 (local address 127.9.242.130:52045)
0927 15:20:39.397485 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.397536 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.398784 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.399277 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.399284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.399409 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:39.399885 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.399897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.400634 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.400636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.401499 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.401508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.402039 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.402042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.402189 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.402723 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:39.402724 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.402764 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:39.402829 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:39.403086 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:39.403091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:39.403095 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.403198 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.403517 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":112,"threads_started":1}
I20260927 15:20:39.405596 30265 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37789
I20260927 15:20:39.405844 30265 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:39.406088 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.242.131:0
--local_ip_for_outbound_sockets=127.9.242.131
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37789
--builtin_ntp_servers=127.9.242.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 15:20:39.406426 30265 heartbeater.cc:511] Master 127.9.242.190:37789 requested a full tablet report, sending...
I20260927 15:20:39.407467 29946 ts_manager.cc:194] Registered new tserver with Master: fde29fbc9a8a49ea95c9a53585630a79 (127.9.242.130:39223)
I20260927 15:20:39.408000 29946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.130:52045
W20260927 15:20:39.512800 30268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:39.513051 30268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:39.513103 30268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:39.517107 30268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:39.517165 30268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:39.517238 30268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.242.131
I20260927 15:20:39.522049 30268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.242.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.242.131
--webserver_port=0
--tserver_master_addrs=127.9.242.190:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.242.131
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:39.523067 30268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:39.524379 30268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:39.533675 30273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:39.533816 30268 server_base.cc:1034] running on GCE node
W20260927 15:20:39.533739 30274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:39.534081 30276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:39.534482 30268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:39.535148 30268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:39.536319 30268 hybrid_clock.cc:648] HybridClock initialized: now 1790522439536299 us; error 34 us; skew 500 ppm
I20260927 15:20:39.538153 30268 webserver.cc:460] Webserver started at http://127.9.242.131:34191/ using document root <none> and password file <none>
I20260927 15:20:39.538714 30268 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:39.538789 30268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:39.538976 30268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:39.540946 30268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data/instance:
uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "80ff7473ea983b9101df06af0f8fbc6a"
server_key_iv: "0cce17af557fb4bf3472613700300edd"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.541481 30268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance:
uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "80ff7473ea983b9101df06af0f8fbc6a"
server_key_iv: "0cce17af557fb4bf3472613700300edd"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.545004 30268 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:39.547554 30282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.548807 30268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 15:20:39.548918 30268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal
uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111"
format_stamp: "Formatted at 2026-09-27 15:20:39 on dist-test-slave-m4lt"
server_key: "80ff7473ea983b9101df06af0f8fbc6a"
server_key_iv: "0cce17af557fb4bf3472613700300edd"
server_key_version: "encryptionkey@0"
I20260927 15:20:39.549021 30268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:39.574580 30268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:39.575353 30268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:39.575536 30268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:39.576115 30268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 15:20:39.577060 30268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 15:20:39.577105 30268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.577157 30268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 15:20:39.577183 30268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:39.585539 30268 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.131:37651
I20260927 15:20:39.585561 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.131:37651 every 8 connection(s)
I20260927 15:20:39.586457 30268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/data/info.pb
I20260927 15:20:39.592089 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30268
I20260927 15:20:39.592186 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal/instance
I20260927 15:20:39.592428 10186 external_mini_cluster.cc:1428] Setting key aad55e59c0b211bb2bf52c8525a59640
I20260927 15:20:39.595549 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.588305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:44063 (local address 127.9.242.190:37789)
0927 15:20:39.588445 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.588449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.589589 (+ 1140us) server_negotiation.cc:407] Connection header received
0927 15:20:39.590986 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.590988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.591034 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.591106 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.592058 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.592710 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.593842 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.594055 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.594140 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.595069 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:39.595072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:39.595074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:39.595084 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:39.595101 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:39.595109 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:39.595165 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.595342 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.595401 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 15:20:39.596355 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.588817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:37789 (local address 127.9.242.131:44063)
0927 15:20:39.589464 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.589505 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.590782 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.591253 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.591259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.591381 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:20:39.591914 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.591931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.592828 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.592832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.593740 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.593749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.594181 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.594185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.594342 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.594837 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 15:20:39.594838 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:20:39.594877 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 15:20:39.594938 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:20:39.595175 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:20:39.595182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:20:39.595188 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.595302 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.595568 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260927 15:20:39.597390 30396 heartbeater.cc:347] Connected to a master server at 127.9.242.190:37789
I20260927 15:20:39.597599 30396 heartbeater.cc:464] Registering TS with master...
I20260927 15:20:39.598117 30396 heartbeater.cc:511] Master 127.9.242.190:37789 requested a full tablet report, sending...
I20260927 15:20:39.599038 29946 ts_manager.cc:194] Registered new tserver with Master: ae9f4f39b5f84bc3a67e0f3ea5bca111 (127.9.242.131:37651)
I20260927 15:20:39.599574 29946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.242.131:44063
I20260927 15:20:39.605785 10186 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 15:20:39.611043 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.607612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.9.242.190:37789)
0927 15:20:39.607738 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.607742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.607874 (+ 132us) server_negotiation.cc:407] Connection header received
0927 15:20:39.607989 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.607992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.608033 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.608094 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:39.608348 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.608850 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.609673 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.609895 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.609981 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.610682 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:39.610687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:39.610689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:39.610699 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:39.610715 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:39.610722 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:39.610786 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.610866 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.610915 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:20:39.613634 29946 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 15:20:39.616583 29946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 15:20:39.630587 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.625569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.9.242.131:37651)
0927 15:20:39.626099 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.626111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.626145 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:20:39.626218 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.626223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.626414 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.626560 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:39.626685 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.627833 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.629382 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.629960 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.630056 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.630245 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.630340 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.630433 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":115,"threads_started":1}
I20260927 15:20:39.630584 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.625043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.131:37651 (local address 127.0.0.1:35936)
0927 15:20:39.625608 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.625620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.625727 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.626534 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.626537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.626548 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:39.626620 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.626625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.627936 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.627939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.629258 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.629270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.630067 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.630070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.630173 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.630210 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.630313 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.630430 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":57,"thread_start_us":80,"threads_started":1}
I20260927 15:20:39.630890 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.624815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:39223 (local address 127.0.0.1:45630)
0927 15:20:39.625149 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.625194 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.625314 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.626445 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.626451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.626472 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:39.626612 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.626618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.627793 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.627796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.629277 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.629283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.630378 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.630382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.630468 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.630490 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.630538 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.630609 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":74,"thread_start_us":85,"threads_started":1}
I20260927 15:20:39.631301 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.625529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36935 (local address 127.0.0.1:42344)
0927 15:20:39.626334 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.626347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.626421 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.627023 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.627026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.627045 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:39.627131 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.627137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.628274 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.628277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.629321 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.629332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.630576 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.630580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.630653 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.630889 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.631059 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.631126 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":42,"thread_start_us":92,"threads_started":1}
I20260927 15:20:39.631536 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.624962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.9.242.130:39223)
0927 15:20:39.625929 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.625941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.625967 (+ 26us) server_negotiation.cc:407] Connection header received
0927 15:20:39.626047 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.626053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.626268 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.626754 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:39.626860 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.627678 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.629416 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.630242 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.631062 (+ 820us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.631237 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.631289 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.631382 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":155,"threads_started":1}
I20260927 15:20:39.631590 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.625848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.9.242.129:36935)
0927 15:20:39.626598 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.626609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.626631 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:20:39.626701 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.626705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.626880 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.627018 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:39.627231 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.628180 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.629442 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.630245 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.630346 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.630523 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.631175 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.631272 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":143,"threads_started":1}
I20260927 15:20:39.633754 30330 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5bbae61f2345dc80e13a026a8af112 (DEFAULT_TABLE table=test-table [id=38f6f18e3dd141be9d7f4b26408eb283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:39.633951 30199 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5bbae61f2345dc80e13a026a8af112 (DEFAULT_TABLE table=test-table [id=38f6f18e3dd141be9d7f4b26408eb283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:39.634111 30068 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5bbae61f2345dc80e13a026a8af112 (DEFAULT_TABLE table=test-table [id=38f6f18e3dd141be9d7f4b26408eb283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 15:20:39.634840 30199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5bbae61f2345dc80e13a026a8af112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:39.634847 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5bbae61f2345dc80e13a026a8af112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:39.634956 30068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5bbae61f2345dc80e13a026a8af112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:39.641719 30413 tablet_bootstrap.cc:492] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Bootstrap starting.
I20260927 15:20:39.642297 30415 tablet_bootstrap.cc:492] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: Bootstrap starting.
I20260927 15:20:39.643713 30413 tablet_bootstrap.cc:654] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:39.643944 30415 tablet_bootstrap.cc:654] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:39.644410 30413 log.cc:824] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:39.644569 30415 log.cc:824] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:39.645035 30414 tablet_bootstrap.cc:492] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Bootstrap starting.
I20260927 15:20:39.646400 30413 tablet_bootstrap.cc:492] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: No bootstrap required, opened a new log
I20260927 15:20:39.646400 30415 tablet_bootstrap.cc:492] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: No bootstrap required, opened a new log
I20260927 15:20:39.646593 30413 ts_tablet_manager.cc:1397] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 15:20:39.646598 30415 ts_tablet_manager.cc:1397] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 15:20:39.646900 30414 tablet_bootstrap.cc:654] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:39.647604 30414 log.cc:824] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:39.649271 30414 tablet_bootstrap.cc:492] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: No bootstrap required, opened a new log
I20260927 15:20:39.649446 30414 ts_tablet_manager.cc:1397] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 15:20:39.650856 30413 raft_consensus.cc:348] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.650856 30415 raft_consensus.cc:348] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.651125 30415 raft_consensus.cc:374] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:39.651125 30413 raft_consensus.cc:374] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:39.651180 30415 raft_consensus.cc:729] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607387fd7869437bb51c8f2245f4e21e, State: Initialized, Role: FOLLOWER
I20260927 15:20:39.651180 30413 raft_consensus.cc:729] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde29fbc9a8a49ea95c9a53585630a79, State: Initialized, Role: FOLLOWER
I20260927 15:20:39.651634 30413 consensus_queue.cc:260] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.651634 30415 consensus_queue.cc:260] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.652277 30134 heartbeater.cc:503] Master 127.9.242.190:37789 was elected leader, sending a full tablet report...
I20260927 15:20:39.652681 30415 ts_tablet_manager.cc:1428] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 15:20:39.652908 30414 raft_consensus.cc:348] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.653126 30414 raft_consensus.cc:374] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:39.653179 30414 raft_consensus.cc:729] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9f4f39b5f84bc3a67e0f3ea5bca111, State: Initialized, Role: FOLLOWER
I20260927 15:20:39.653589 30414 consensus_queue.cc:260] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.653702 30413 ts_tablet_manager.cc:1428] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 15:20:39.654255 30396 heartbeater.cc:503] Master 127.9.242.190:37789 was elected leader, sending a full tablet report...
I20260927 15:20:39.654551 30414 ts_tablet_manager.cc:1428] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 15:20:39.654897 30265 heartbeater.cc:503] Master 127.9.242.190:37789 was elected leader, sending a full tablet report...
I20260927 15:20:39.690953 30421 raft_consensus.cc:482] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 15:20:39.691143 30421 raft_consensus.cc:504] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.692466 30421 leader_election.cc:288] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde29fbc9a8a49ea95c9a53585630a79 (127.9.242.130:39223), 607387fd7869437bb51c8f2245f4e21e (127.9.242.129:36935)
I20260927 15:20:39.695819 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.692646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.130:39223 (local address 127.9.242.131:55043)
0927 15:20:39.692803 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.692819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.692915 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.693194 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.693197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.693215 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:39.693313 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.693318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.694056 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.694060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.694822 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.694830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.695438 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.695441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.695534 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.695556 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.695619 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.695686 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 15:20:39.695818 30408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.692766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:55043 (local address 127.9.242.130:39223)
0927 15:20:39.692899 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.692903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.692917 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:20:39.693003 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.693006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.693052 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.693116 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:39.693417 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.693956 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.694933 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.695340 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.695438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.695488 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.695644 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.695708 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 15:20:39.696200 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.692886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36935 (local address 127.9.242.131:49265)
0927 15:20:39.693207 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.693218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 15:20:39.693311 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:20:39.693517 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:20:39.693520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:20:39.693533 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 15:20:39.693664 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.693670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.694338 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.694342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.695084 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:20:39.695092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.695787 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.695791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:20:39.695881 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.695904 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 15:20:39.696008 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 15:20:39.696068 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260927 15:20:39.696233 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.693000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.242.131:49265 (local address 127.9.242.129:36935)
0927 15:20:39.693134 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.693139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.693300 (+ 161us) server_negotiation.cc:407] Connection header received
0927 15:20:39.693392 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.693396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.693439 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.693497 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 15:20:39.693764 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.694241 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.695195 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.695670 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.695773 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.695817 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.696042 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.696122 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 15:20:39.696471 30219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5bbae61f2345dc80e13a026a8af112" candidate_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde29fbc9a8a49ea95c9a53585630a79" is_pre_election: true
I20260927 15:20:39.696728 30219 raft_consensus.cc:2454] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9f4f39b5f84bc3a67e0f3ea5bca111 in term 0.
I20260927 15:20:39.697150 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5bbae61f2345dc80e13a026a8af112" candidate_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607387fd7869437bb51c8f2245f4e21e" is_pre_election: true
I20260927 15:20:39.697196 30285 leader_election.cc:302] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9f4f39b5f84bc3a67e0f3ea5bca111, fde29fbc9a8a49ea95c9a53585630a79; no voters:
I20260927 15:20:39.697449 30421 raft_consensus.cc:2781] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 15:20:39.697510 30088 raft_consensus.cc:2454] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9f4f39b5f84bc3a67e0f3ea5bca111 in term 0.
I20260927 15:20:39.697544 30421 raft_consensus.cc:482] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 15:20:39.697587 30421 raft_consensus.cc:3037] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:39.698659 30421 raft_consensus.cc:504] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.698971 30421 leader_election.cc:288] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [CANDIDATE]: Term 1 election: Requested vote from peers fde29fbc9a8a49ea95c9a53585630a79 (127.9.242.130:39223), 607387fd7869437bb51c8f2245f4e21e (127.9.242.129:36935)
I20260927 15:20:39.699317 30088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5bbae61f2345dc80e13a026a8af112" candidate_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607387fd7869437bb51c8f2245f4e21e"
I20260927 15:20:39.699311 30219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5bbae61f2345dc80e13a026a8af112" candidate_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde29fbc9a8a49ea95c9a53585630a79"
I20260927 15:20:39.699430 30088 raft_consensus.cc:3037] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:39.699434 30219 raft_consensus.cc:3037] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:39.700625 30219 raft_consensus.cc:2454] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9f4f39b5f84bc3a67e0f3ea5bca111 in term 1.
I20260927 15:20:39.700625 30088 raft_consensus.cc:2454] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9f4f39b5f84bc3a67e0f3ea5bca111 in term 1.
I20260927 15:20:39.700894 30285 leader_election.cc:302] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9f4f39b5f84bc3a67e0f3ea5bca111, fde29fbc9a8a49ea95c9a53585630a79; no voters:
I20260927 15:20:39.701040 30421 raft_consensus.cc:2781] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:39.701223 30421 raft_consensus.cc:686] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 1 LEADER]: Becoming Leader. State: Replica: ae9f4f39b5f84bc3a67e0f3ea5bca111, State: Running, Role: LEADER
I20260927 15:20:39.701524 30421 consensus_queue.cc:237] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } }
I20260927 15:20:39.704598 29946 catalog_manager.cc:5495] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9f4f39b5f84bc3a67e0f3ea5bca111 (127.9.242.131). New cstate: current_term: 1 leader_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9f4f39b5f84bc3a67e0f3ea5bca111" member_type: VOTER last_known_addr { host: "127.9.242.131" port: 37651 } health_report { overall_health: HEALTHY } } }
W20260927 15:20:39.706753 30135 tablet.cc:2369] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:20:39.729849 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.726614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.9.242.131:37651)
0927 15:20:39.726721 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:39.726726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:39.726738 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:20:39.726868 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:39.726870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:39.726922 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:39.727014 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:20:39.727202 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.727703 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.728568 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:39.728767 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:39.728848 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:39.729067 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:20:39.729466 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:20:39.729549 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:39.729597 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 15:20:39.729640 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:20:39.735427 30088 raft_consensus.cc:1261] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 1 FOLLOWER]: Refusing update from remote peer ae9f4f39b5f84bc3a67e0f3ea5bca111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:39.735427 30219 raft_consensus.cc:1261] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 1 FOLLOWER]: Refusing update from remote peer ae9f4f39b5f84bc3a67e0f3ea5bca111: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:20:39.736148 30421 consensus_queue.cc:1035] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde29fbc9a8a49ea95c9a53585630a79" member_type: VOTER last_known_addr { host: "127.9.242.130" port: 39223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:39.736394 30423 consensus_queue.cc:1035] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607387fd7869437bb51c8f2245f4e21e" member_type: VOTER last_known_addr { host: "127.9.242.129" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:20:39.753355 29946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:39.753504 29946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 15:20:39.755695 29946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc8469a2ca8345aa8bac82e6ff10b73c: Sending DeleteTablet for 3 replicas of tablet 3c5bbae61f2345dc80e13a026a8af112
I20260927 15:20:39.756434 30199 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5bbae61f2345dc80e13a026a8af112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:39 UTC) from {username='slave'} at 127.0.0.1:45630
I20260927 15:20:39.756433 30068 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5bbae61f2345dc80e13a026a8af112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:39 UTC) from {username='slave'} at 127.0.0.1:42344
I20260927 15:20:39.756898 30330 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c5bbae61f2345dc80e13a026a8af112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 15:20:39 UTC) from {username='slave'} at 127.0.0.1:35936
I20260927 15:20:39.757045 30437 tablet_replica.cc:330] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e: stopping tablet replica
I20260927 15:20:39.757246 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30006
I20260927 15:20:39.757355 30436 tablet_replica.cc:330] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: stopping tablet replica
I20260927 15:20:39.757351 30437 raft_consensus.cc:2229] T 3c5bbae61f2345dc80e13a026a8af112 P 607387fd7869437bb51c8f2245f4e21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:39.757368 30438 tablet_replica.cc:330] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: stopping tablet replica
I20260927 15:20:39.757602 30436 raft_consensus.cc:2229] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 15:20:39.757789 30438 raft_consensus.cc:2229] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 1 LEADER]: Raft consensus shutting down.
I20260927 15:20:39.757884 30436 raft_consensus.cc:2258] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:39.758172 30438 raft_consensus.cc:2258] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 15:20:39.759516 30436 ts_tablet_manager.cc:1905] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:39.759742 30438 ts_tablet_manager.cc:1905] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 15:20:39.762223 30438 ts_tablet_manager.cc:1918] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:39.762321 30438 log.cc:1196] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-2/wal/wals/3c5bbae61f2345dc80e13a026a8af112
I20260927 15:20:39.762465 30436 ts_tablet_manager.cc:1918] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 15:20:39.762584 30438 ts_tablet_manager.cc:1939] T 3c5bbae61f2345dc80e13a026a8af112 P ae9f4f39b5f84bc3a67e0f3ea5bca111: Deleting consensus metadata
I20260927 15:20:39.762683 30436 log.cc:1196] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Deleting WAL directory at /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790522194092531-10186-0/minicluster-data/ts-1/wal/wals/3c5bbae61f2345dc80e13a026a8af112
I20260927 15:20:39.763034 30436 ts_tablet_manager.cc:1939] T 3c5bbae61f2345dc80e13a026a8af112 P fde29fbc9a8a49ea95c9a53585630a79: Deleting consensus metadata
I20260927 15:20:39.763700 29933 catalog_manager.cc:4841] TS ae9f4f39b5f84bc3a67e0f3ea5bca111 (127.9.242.131:37651): tablet 3c5bbae61f2345dc80e13a026a8af112 (table test-table [id=38f6f18e3dd141be9d7f4b26408eb283]) successfully deleted
I20260927 15:20:39.764274 29932 catalog_manager.cc:4841] TS fde29fbc9a8a49ea95c9a53585630a79 (127.9.242.130:39223): tablet 3c5bbae61f2345dc80e13a026a8af112 (table test-table [id=38f6f18e3dd141be9d7f4b26408eb283]) successfully deleted
W20260927 15:20:39.764648 29932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.242.129:36935 (error 108)
I20260927 15:20:39.765453 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30137
I20260927 15:20:39.765673 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:39.765380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.129:36935 (local address 127.0.0.1:42358)
0927 15:20:39.765496 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 15:20:39.765567 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.129:36935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 15:20:39.766734 29932 catalog_manager.cc:4584] TS 607387fd7869437bb51c8f2245f4e21e (127.9.242.129:36935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c5bbae61f2345dc80e13a026a8af112: Network error: Client connection negotiation failed: client connection to 127.9.242.129:36935: connect: Connection refused (error 111)
I20260927 15:20:39.770674 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30268
I20260927 15:20:39.775192 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 29915
2026-09-27T15:20:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1031 ms)
[----------] 6 tests from EncryptionPolicyTest (10289 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:39 dist-test-slave-m4lt krb5kdc[30442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:39 dist-test-slave-m4lt krb5kdc[30442](info): set up 2 sockets
Sep 27 15:20:39 dist-test-slave-m4lt krb5kdc[30442](info): commencing operation
krb5kdc: starting...
W20260927 15:20:40.729485 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:40 dist-test-slave-m4lt krb5kdc[30442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:40.846653 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:40.954672 30455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:40.954929 30455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:40.954972 30455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:40.958914 30455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:40.958971 30455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:40.958988 30455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:40.959005 30455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:40.959033 30455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:40.964262 30455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:40.965303 30455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:40.966333 30455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:40.974278 30460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:40.974341 30463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:40.974357 30461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:40.974453 30455 server_base.cc:1034] running on GCE node
I20260927 15:20:40.975070 30455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:40.976147 30455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:40.977299 30455 hybrid_clock.cc:648] HybridClock initialized: now 1790522440977285 us; error 32 us; skew 500 ppm
Sep 27 15:20:40 dist-test-slave-m4lt krb5kdc[30442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:40.979930 30455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:40.980794 30455 webserver.cc:460] Webserver started at http://127.0.0.1:34581/ using document root <none> and password file <none>
I20260927 15:20:40.981310 30455 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:40.981374 30455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:40.981550 30455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:40.983541 30455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "864cadc8a6fb40f49fbafc8829d5ba22"
format_stamp: "Formatted at 2026-09-27 15:20:40 on dist-test-slave-m4lt"
server_key: "9c72f870e6253cef21cfc2046d09c136"
server_key_iv: "84afd997769b59381925c5ea8815d078"
server_key_version: "encryptionkey@0"
I20260927 15:20:40.984059 30455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "864cadc8a6fb40f49fbafc8829d5ba22"
format_stamp: "Formatted at 2026-09-27 15:20:40 on dist-test-slave-m4lt"
server_key: "9c72f870e6253cef21cfc2046d09c136"
server_key_iv: "84afd997769b59381925c5ea8815d078"
server_key_version: "encryptionkey@0"
I20260927 15:20:40.987536 30455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:20:40.989748 30470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:40.990957 30455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:40.991066 30455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "864cadc8a6fb40f49fbafc8829d5ba22"
format_stamp: "Formatted at 2026-09-27 15:20:40 on dist-test-slave-m4lt"
server_key: "9c72f870e6253cef21cfc2046d09c136"
server_key_iv: "84afd997769b59381925c5ea8815d078"
server_key_version: "encryptionkey@0"
I20260927 15:20:40.991156 30455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:41.014295 30455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:41.017515 30455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:41.017733 30455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:41.024653 30455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37851
I20260927 15:20:41.024709 30522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37851 every 8 connection(s)
I20260927 15:20:41.025604 30455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:41.028399 30523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:41.032539 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30455
I20260927 15:20:41.032671 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:41.032927 10186 external_mini_cluster.cc:1428] Setting key b658d25acc0f16c50be5e82e4723eb1c
I20260927 15:20:41.034313 30523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22: Bootstrap starting.
I20260927 15:20:41.036616 30523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:41.037343 30523 log.cc:824] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:41.039397 30523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22: No bootstrap required, opened a new log
Sep 27 15:20:41 dist-test-slave-m4lt krb5kdc[30442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:20:41.043299 30523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37851 } }
I20260927 15:20:41.043511 30523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:41.043545 30523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864cadc8a6fb40f49fbafc8829d5ba22, State: Initialized, Role: FOLLOWER
I20260927 15:20:41.043967 30523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37851 } }
I20260927 15:20:41.044082 30523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:41.044139 30523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:41.044225 30523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:41.045054 30523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37851 } }
I20260927 15:20:41.045378 30523 leader_election.cc:302] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 864cadc8a6fb40f49fbafc8829d5ba22; no voters:
I20260927 15:20:41.045670 30523 leader_election.cc:288] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:41.045817 30528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:41.046026 30528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [term 1 LEADER]: Becoming Leader. State: Replica: 864cadc8a6fb40f49fbafc8829d5ba22, State: Running, Role: LEADER
I20260927 15:20:41.046370 30528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37851 } }
I20260927 15:20:41.046947 30523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:41.047163 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:41.034277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52889 (local address 127.0.0.1:37851)
0927 15:20:41.034671 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:41.034687 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:41.034722 (+ 35us) server_negotiation.cc:407] Connection header received
0927 15:20:41.035343 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:41.035360 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:41.035693 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:41.035970 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:41.036817 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.037970 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.039138 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.039639 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.039747 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:41.042390 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:41.042394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:41.042408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:41.042433 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:41.044099 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:41.044608 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:41.044614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:41.044619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:41.044725 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:41.045031 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:41.045035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:41.045036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:41.045484 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:41.045717 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:41.046023 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 15:20:41.046377 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260927 15:20:41.047523 30528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37851 } } }
I20260927 15:20:41.047631 30528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:41.047586 30529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 864cadc8a6fb40f49fbafc8829d5ba22. Latest consensus state: current_term: 1 leader_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "864cadc8a6fb40f49fbafc8829d5ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37851 } } }
I20260927 15:20:41.047686 30529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:41.050472 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 15:20:41.051733 30543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:41.051824 30543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:41.051892 30534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:41.052726 30534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:41.058341 30534 catalog_manager.cc:1347] Generated new cluster ID: 912fa3bc02304d6683e1daaf85c416f3
I20260927 15:20:41.058408 30534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:41.062548 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:41.052591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44979 (local address 127.0.0.1:37851)
0927 15:20:41.052701 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:41.052705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:41.054511 (+ 1806us) server_negotiation.cc:407] Connection header received
0927 15:20:41.054626 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:41.054630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:41.054680 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:41.054767 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:41.055521 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.056413 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.057252 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:41.057505 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:41.057592 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:41.059154 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:41.059158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:41.059161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:41.059188 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:41.060826 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:41.061317 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:41.061320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:41.061322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:41.061407 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:41.061715 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:41.061718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:41.061720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:41.061923 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:41.062063 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:41.062301 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 15:20:41.062393 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 15:20:41.078411 30534 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:41.079548 30534 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:41.086040 30534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 864cadc8a6fb40f49fbafc8829d5ba22: Generated new TSK 0
I20260927 15:20:41.086740 30534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:41.101883 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30455
W20260927 15:20:41.106519 30440 connection.cc:535] client connection to 127.0.0.1:37851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37851): Connection reset by peer (error 104)
2026-09-27T15:20:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:41 dist-test-slave-m4lt krb5kdc[30554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:41 dist-test-slave-m4lt krb5kdc[30554](info): set up 2 sockets
Sep 27 15:20:41 dist-test-slave-m4lt krb5kdc[30554](info): commencing operation
krb5kdc: starting...
W20260927 15:20:42.057644 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:42 dist-test-slave-m4lt krb5kdc[30554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:42.173000 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:42.280439 30567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:42.280735 30567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:42.280813 30567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:42.285023 30567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:42.285087 30567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:42.285115 30567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:42.285136 30567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:42.285161 30567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:42.290465 30567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:42.291515 30567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:42.292505 30567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:42.300977 30572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:42.301186 30567 server_base.cc:1034] running on GCE node
W20260927 15:20:42.300977 30573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:42.301035 30575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:42.301887 30567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:42.303040 30567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:42.304210 30567 hybrid_clock.cc:648] HybridClock initialized: now 1790522442304184 us; error 37 us; skew 500 ppm
Sep 27 15:20:42 dist-test-slave-m4lt krb5kdc[30554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:42.307051 30567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:42.307935 30567 webserver.cc:460] Webserver started at http://127.0.0.1:46411/ using document root <none> and password file <none>
I20260927 15:20:42.308486 30567 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:42.308547 30567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:42.308743 30567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:42.310835 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "923cc7ee2e6c4de58f37b73dbf3570d7"
format_stamp: "Formatted at 2026-09-27 15:20:42 on dist-test-slave-m4lt"
server_key: "2ba79872e55b4cb0ee019dd5ad1c42dc"
server_key_iv: "78410ba2b09ec56103406cab64011a22"
server_key_version: "encryptionkey@0"
I20260927 15:20:42.311375 30567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "923cc7ee2e6c4de58f37b73dbf3570d7"
format_stamp: "Formatted at 2026-09-27 15:20:42 on dist-test-slave-m4lt"
server_key: "2ba79872e55b4cb0ee019dd5ad1c42dc"
server_key_iv: "78410ba2b09ec56103406cab64011a22"
server_key_version: "encryptionkey@0"
I20260927 15:20:42.315143 30567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:20:42.317528 30582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:42.318831 30567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:42.318954 30567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "923cc7ee2e6c4de58f37b73dbf3570d7"
format_stamp: "Formatted at 2026-09-27 15:20:42 on dist-test-slave-m4lt"
server_key: "2ba79872e55b4cb0ee019dd5ad1c42dc"
server_key_iv: "78410ba2b09ec56103406cab64011a22"
server_key_version: "encryptionkey@0"
I20260927 15:20:42.319060 30567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:42.351949 30567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:42.355156 30567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:42.355337 30567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:42.362025 30567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38907
I20260927 15:20:42.362115 30634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38907 every 8 connection(s)
I20260927 15:20:42.362984 30567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:42.365787 30635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:42.369174 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30567
I20260927 15:20:42.369315 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:42.369598 10186 external_mini_cluster.cc:1428] Setting key 018db258cf71669ac42bb7ff873668f6
I20260927 15:20:42.371631 30635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7: Bootstrap starting.
I20260927 15:20:42.374002 30635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:42.374882 30635 log.cc:824] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:42.377139 30635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7: No bootstrap required, opened a new log
I20260927 15:20:42.379153 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:42.370991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36915 (local address 127.0.0.1:38907)
0927 15:20:42.371406 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:42.371425 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:42.371470 (+ 45us) server_negotiation.cc:407] Connection header received
0927 15:20:42.372127 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:42.372145 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:42.372511 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:42.372792 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:42.373115 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.374221 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.375380 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.375977 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.376094 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.376943 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:42.376948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:42.376961 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:42.377527 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:42.377592 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:42.377616 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:42.377804 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:42.377984 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 15:20:42.378120 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":117,"threads_started":1}
I20260927 15:20:42.380245 30635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38907 } }
I20260927 15:20:42.380471 30635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:42.380522 30635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923cc7ee2e6c4de58f37b73dbf3570d7, State: Initialized, Role: FOLLOWER
I20260927 15:20:42.380957 30635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38907 } }
I20260927 15:20:42.381120 30635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:42.381189 30635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:42.381285 30635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:42.382001 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:42.382251 30635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38907 } }
I20260927 15:20:42.382582 30635 leader_election.cc:302] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 923cc7ee2e6c4de58f37b73dbf3570d7; no voters:
I20260927 15:20:42.382866 30635 leader_election.cc:288] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:42.382956 30640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:42.383111 30640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [term 1 LEADER]: Becoming Leader. State: Replica: 923cc7ee2e6c4de58f37b73dbf3570d7, State: Running, Role: LEADER
I20260927 15:20:42.383441 30640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38907 } }
I20260927 15:20:42.384325 30635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:42.385147 30641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 923cc7ee2e6c4de58f37b73dbf3570d7. Latest consensus state: current_term: 1 leader_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38907 } } }
I20260927 15:20:42.385188 30642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "923cc7ee2e6c4de58f37b73dbf3570d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38907 } } }
I20260927 15:20:42.385264 30641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:42.385290 30642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:42.385643 30654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:42.388779 30654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:42.389914 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:42.384090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59507 (local address 127.0.0.1:38907)
0927 15:20:42.384205 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:42.384210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:42.384423 (+ 213us) server_negotiation.cc:407] Connection header received
0927 15:20:42.385512 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:42.385515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:42.385563 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:42.385644 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:42.385918 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.386631 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.387744 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:42.388075 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:42.388158 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:42.389217 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:42.389221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:42.389224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:42.389238 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:42.389267 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:42.389280 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:42.389357 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:42.389749 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 15:20:42.389783 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:20:42.394222 30654 catalog_manager.cc:1347] Generated new cluster ID: 2b58242c1faa48dc87cce1b61c1f6a59
I20260927 15:20:42.394289 30654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:42.410461 30654 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:42.411724 30654 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:42.418206 30654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 923cc7ee2e6c4de58f37b73dbf3570d7: Generated new TSK 0
I20260927 15:20:42.418880 30654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:42.424496 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30567
W20260927 15:20:42.428875 30552 connection.cc:535] client connection to 127.0.0.1:38907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38907): Connection reset by peer (error 104)
2026-09-27T15:20:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:42 dist-test-slave-m4lt krb5kdc[30666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:42 dist-test-slave-m4lt krb5kdc[30666](info): set up 2 sockets
Sep 27 15:20:42 dist-test-slave-m4lt krb5kdc[30666](info): commencing operation
krb5kdc: starting...
W20260927 15:20:43.385874 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:43 dist-test-slave-m4lt krb5kdc[30666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:43.498778 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:43.605083 30679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:43.605373 30679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:43.605436 30679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:43.609416 30679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:43.609483 30679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:43.609508 30679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:43.609530 30679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:43.609555 30679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:43.614720 30679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:43.615705 30679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:43.616659 30679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:43.624781 30684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:43.624785 30685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:43.624835 30687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:43.625000 30679 server_base.cc:1034] running on GCE node
I20260927 15:20:43.625582 30679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:43.626719 30679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:43.627892 30679 hybrid_clock.cc:648] HybridClock initialized: now 1790522443627876 us; error 30 us; skew 500 ppm
Sep 27 15:20:43 dist-test-slave-m4lt krb5kdc[30666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:43.630486 30679 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:43.631295 30679 webserver.cc:460] Webserver started at http://127.0.0.1:37597/ using document root <none> and password file <none>
I20260927 15:20:43.631793 30679 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:43.631847 30679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:43.632005 30679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:43.633980 30679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "6dd807d8e987438fb35ee82773d7edda"
format_stamp: "Formatted at 2026-09-27 15:20:43 on dist-test-slave-m4lt"
server_key: "6a2f2a2bbbc2c75fc6dd6204ddc62d3c"
server_key_iv: "f3916bb9b0d9d5809d631c40bfce6af5"
server_key_version: "encryptionkey@0"
I20260927 15:20:43.634473 30679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "6dd807d8e987438fb35ee82773d7edda"
format_stamp: "Formatted at 2026-09-27 15:20:43 on dist-test-slave-m4lt"
server_key: "6a2f2a2bbbc2c75fc6dd6204ddc62d3c"
server_key_iv: "f3916bb9b0d9d5809d631c40bfce6af5"
server_key_version: "encryptionkey@0"
I20260927 15:20:43.638022 30679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:20:43.640556 30694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:43.641884 30679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:43.642009 30679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "6dd807d8e987438fb35ee82773d7edda"
format_stamp: "Formatted at 2026-09-27 15:20:43 on dist-test-slave-m4lt"
server_key: "6a2f2a2bbbc2c75fc6dd6204ddc62d3c"
server_key_iv: "f3916bb9b0d9d5809d631c40bfce6af5"
server_key_version: "encryptionkey@0"
I20260927 15:20:43.642098 30679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:43.646443 30679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:43.649348 30679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:43.649502 30679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:43.656010 30679 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40635
I20260927 15:20:43.656037 30746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40635 every 8 connection(s)
I20260927 15:20:43.656913 30679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:43.659593 30747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:43.664354 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30679
I20260927 15:20:43.664479 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:43.664727 10186 external_mini_cluster.cc:1428] Setting key 4005000191e8ed75ecf7482ef7ec0716
I20260927 15:20:43.665270 30747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda: Bootstrap starting.
I20260927 15:20:43.667797 30747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:43.668617 30747 log.cc:824] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:43.669925 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:43.666149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40001 (local address 127.0.0.1:40635)
0927 15:20:43.666603 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:43.666617 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:43.666654 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:20:43.667343 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:43.667361 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:43.667644 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:43.667950 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:43.668703 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:43.668709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:43.668725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:43.668736 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:43.668801 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:43.668827 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:43.668920 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:43.669035 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 15:20:43.669184 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260927 15:20:43.670574 30747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda: No bootstrap required, opened a new log
I20260927 15:20:43.672586 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:43.674386 30747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd807d8e987438fb35ee82773d7edda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260927 15:20:43.674674 30747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:43.674729 30747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd807d8e987438fb35ee82773d7edda, State: Initialized, Role: FOLLOWER
I20260927 15:20:43.675237 30747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd807d8e987438fb35ee82773d7edda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260927 15:20:43.675413 30747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:43.675490 30747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:43.675595 30747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:43.676256 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:43.674609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45931 (local address 127.0.0.1:40635)
0927 15:20:43.674741 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:43.674746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:43.675005 (+ 259us) server_negotiation.cc:407] Connection header received
0927 15:20:43.675118 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:43.675120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:43.675151 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:43.675215 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:43.675924 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:43.675927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:43.675929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:43.675941 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:43.675969 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:43.675982 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:43.676069 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:43.676107 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 15:20:43.676140 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 15:20:43.676725 30747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd807d8e987438fb35ee82773d7edda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260927 15:20:43.677065 30747 leader_election.cc:302] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dd807d8e987438fb35ee82773d7edda; no voters:
I20260927 15:20:43.677351 30747 leader_election.cc:288] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:43.677438 30757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:43.677598 30757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [term 1 LEADER]: Becoming Leader. State: Replica: 6dd807d8e987438fb35ee82773d7edda, State: Running, Role: LEADER
I20260927 15:20:43.677939 30757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd807d8e987438fb35ee82773d7edda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } }
I20260927 15:20:43.678601 30747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:43.679471 30759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dd807d8e987438fb35ee82773d7edda. Latest consensus state: current_term: 1 leader_uuid: "6dd807d8e987438fb35ee82773d7edda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd807d8e987438fb35ee82773d7edda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } } }
I20260927 15:20:43.679847 30759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:43.679747 30758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dd807d8e987438fb35ee82773d7edda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dd807d8e987438fb35ee82773d7edda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40635 } } }
I20260927 15:20:43.680156 30758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:43.680259 30766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:43.683224 30766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:43.688434 30766 catalog_manager.cc:1347] Generated new cluster ID: ac31438423d14a20820641d431cc53ec
I20260927 15:20:43.688508 30766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:43.697520 30766 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:43.697916 30766 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:43.702538 30766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dd807d8e987438fb35ee82773d7edda: Generated new TSK 0
I20260927 15:20:43.703190 30766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:43.707762 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30679
2026-09-27T15:20:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:43 dist-test-slave-m4lt krb5kdc[30778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:43 dist-test-slave-m4lt krb5kdc[30778](info): set up 2 sockets
Sep 27 15:20:43 dist-test-slave-m4lt krb5kdc[30778](info): commencing operation
krb5kdc: starting...
W20260927 15:20:44.653601 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:44 dist-test-slave-m4lt krb5kdc[30778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:44.770386 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:44.880326 30791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:44.880609 30791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:44.880650 30791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:44.884728 30791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:44.884793 30791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:44.884811 30791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:44.884826 30791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:44.884840 30791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:44.890312 30791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:44.891393 30791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:44.892386 30791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:44.900604 30799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:44.900606 30797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:44.900604 30796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:44.900822 30791 server_base.cc:1034] running on GCE node
I20260927 15:20:44.901314 30791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:44.902501 30791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:44.903669 30791 hybrid_clock.cc:648] HybridClock initialized: now 1790522444903659 us; error 49 us; skew 500 ppm
Sep 27 15:20:44 dist-test-slave-m4lt krb5kdc[30778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:44.911819 30791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:44.912858 30791 webserver.cc:460] Webserver started at http://127.0.0.1:46669/ using document root <none> and password file <none>
I20260927 15:20:44.913421 30791 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:44.913486 30791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:44.913717 30791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:44.915768 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "448b9cd397ea40e48ffdc306686cc45e"
format_stamp: "Formatted at 2026-09-27 15:20:44 on dist-test-slave-m4lt"
server_key: "d589305646105dd85e2424447de84a45"
server_key_iv: "bd72e6f4441f58a8bb1f2ebbf38d332d"
server_key_version: "encryptionkey@0"
I20260927 15:20:44.916292 30791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "448b9cd397ea40e48ffdc306686cc45e"
format_stamp: "Formatted at 2026-09-27 15:20:44 on dist-test-slave-m4lt"
server_key: "d589305646105dd85e2424447de84a45"
server_key_iv: "bd72e6f4441f58a8bb1f2ebbf38d332d"
server_key_version: "encryptionkey@0"
I20260927 15:20:44.919947 30791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 15:20:44.922643 30806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:44.924240 30791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:44.924365 30791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "448b9cd397ea40e48ffdc306686cc45e"
format_stamp: "Formatted at 2026-09-27 15:20:44 on dist-test-slave-m4lt"
server_key: "d589305646105dd85e2424447de84a45"
server_key_iv: "bd72e6f4441f58a8bb1f2ebbf38d332d"
server_key_version: "encryptionkey@0"
I20260927 15:20:44.924471 30791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:44.941092 30791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:44.943866 30791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:44.944028 30791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:44.950488 30791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35353
I20260927 15:20:44.950539 30858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35353 every 8 connection(s)
I20260927 15:20:44.951380 30791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:44.954046 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:44.956408 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30791
I20260927 15:20:44.956532 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:44.956782 10186 external_mini_cluster.cc:1428] Setting key ffa31a7c6c3a77f2740e0e6e57c2606f
I20260927 15:20:44.960064 30859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e: Bootstrap starting.
I20260927 15:20:44.962594 30859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:44.963385 30859 log.cc:824] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:44.965324 30859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e: No bootstrap required, opened a new log
Sep 27 15:20:44 dist-test-slave-m4lt krb5kdc[30778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:20:44.969132 30859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448b9cd397ea40e48ffdc306686cc45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260927 15:20:44.969394 30859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:44.969453 30859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448b9cd397ea40e48ffdc306686cc45e, State: Initialized, Role: FOLLOWER
I20260927 15:20:44.969964 30859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448b9cd397ea40e48ffdc306686cc45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260927 15:20:44.970121 30859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:44.970187 30859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:44.970280 30859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:44.971354 30859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448b9cd397ea40e48ffdc306686cc45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260927 15:20:44.971695 30859 leader_election.cc:302] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 448b9cd397ea40e48ffdc306686cc45e; no voters:
I20260927 15:20:44.972029 30859 leader_election.cc:288] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:44.972116 30864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:44.972278 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:44.958093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55087 (local address 127.0.0.1:35353)
0927 15:20:44.958565 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:44.958582 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:44.958624 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:20:44.959425 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:44.959447 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:44.959859 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:44.960206 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:44.960841 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.962320 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.963249 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.963897 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.964009 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.966590 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:44.966596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:44.966612 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:44.966643 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:44.969142 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:44.969607 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:44.969613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:44.969618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:44.969779 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:44.970027 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:44.970030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:44.970032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:44.970552 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:44.970750 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:44.971003 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 15:20:44.971316 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":167,"threads_started":1}
I20260927 15:20:44.972393 30864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [term 1 LEADER]: Becoming Leader. State: Replica: 448b9cd397ea40e48ffdc306686cc45e, State: Running, Role: LEADER
I20260927 15:20:44.972703 30864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448b9cd397ea40e48ffdc306686cc45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260927 15:20:44.973229 30859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:44.974222 30865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "448b9cd397ea40e48ffdc306686cc45e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448b9cd397ea40e48ffdc306686cc45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } } }
I20260927 15:20:44.974344 30865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:44.974604 30866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448b9cd397ea40e48ffdc306686cc45e. Latest consensus state: current_term: 1 leader_uuid: "448b9cd397ea40e48ffdc306686cc45e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448b9cd397ea40e48ffdc306686cc45e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } } }
I20260927 15:20:44.974677 30866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:44.975523 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 15:20:44.977610 30879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:44.977710 30879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:44.977782 30873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:44.978591 30873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:44.983934 30873 catalog_manager.cc:1347] Generated new cluster ID: 13195123061d4a8aa67d4bde6ca85490
I20260927 15:20:44.983995 30873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:44.987797 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:44.977491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55349 (local address 127.0.0.1:35353)
0927 15:20:44.977591 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:44.977595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:44.978044 (+ 449us) server_negotiation.cc:407] Connection header received
0927 15:20:44.978133 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:44.978135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:44.978179 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:44.978262 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:44.979069 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.979830 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.980997 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:44.981330 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:44.981429 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:44.982942 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:44.982946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:44.982948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:44.982974 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:44.986188 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:44.986564 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:44.986567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:44.986569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:44.986639 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:44.986878 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:44.986881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:44.986882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:44.987041 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:44.987150 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:44.987541 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 15:20:44.987645 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:20:45.004360 30873 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:45.005695 30873 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:45.018885 30873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 448b9cd397ea40e48ffdc306686cc45e: Generated new TSK 0
I20260927 15:20:45.019767 30873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:45.034718 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30791
2026-09-27T15:20:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:45 dist-test-slave-m4lt krb5kdc[30890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:45 dist-test-slave-m4lt krb5kdc[30890](info): set up 2 sockets
Sep 27 15:20:45 dist-test-slave-m4lt krb5kdc[30890](info): commencing operation
krb5kdc: starting...
W20260927 15:20:45.985620 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:46 dist-test-slave-m4lt krb5kdc[30890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:46.102921 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:46.210512 30903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:46.210788 30903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:46.210829 30903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:46.214823 30903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:46.214885 30903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:46.214905 30903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:46.214919 30903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:46.214933 30903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:46.220180 30903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:46.221175 30903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:46.222134 30903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:46.230495 30911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:46.230499 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:46.230610 30903 server_base.cc:1034] running on GCE node
W20260927 15:20:46.230824 30908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:46.231391 30903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:46.232509 30903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:46.233704 30903 hybrid_clock.cc:648] HybridClock initialized: now 1790522446233684 us; error 32 us; skew 500 ppm
Sep 27 15:20:46 dist-test-slave-m4lt krb5kdc[30890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:46.236410 30903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:46.237331 30903 webserver.cc:460] Webserver started at http://127.0.0.1:37071/ using document root <none> and password file <none>
I20260927 15:20:46.237903 30903 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:46.237963 30903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:46.238178 30903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:46.240163 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "b42bf320403b49c59e8ef09d02b0ca81"
format_stamp: "Formatted at 2026-09-27 15:20:46 on dist-test-slave-m4lt"
server_key: "a5d6f6e7a1324ef7a7198a43a8d82fb6"
server_key_iv: "585230cde66ce156f130648ce99ef35b"
server_key_version: "encryptionkey@0"
I20260927 15:20:46.240687 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "b42bf320403b49c59e8ef09d02b0ca81"
format_stamp: "Formatted at 2026-09-27 15:20:46 on dist-test-slave-m4lt"
server_key: "a5d6f6e7a1324ef7a7198a43a8d82fb6"
server_key_iv: "585230cde66ce156f130648ce99ef35b"
server_key_version: "encryptionkey@0"
I20260927 15:20:46.244343 30903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:20:46.246759 30918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:46.248041 30903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:20:46.248155 30903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "b42bf320403b49c59e8ef09d02b0ca81"
format_stamp: "Formatted at 2026-09-27 15:20:46 on dist-test-slave-m4lt"
server_key: "a5d6f6e7a1324ef7a7198a43a8d82fb6"
server_key_iv: "585230cde66ce156f130648ce99ef35b"
server_key_version: "encryptionkey@0"
I20260927 15:20:46.248324 30903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:46.264653 30903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:46.267477 30903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:46.267652 30903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:46.274459 30903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37209
I20260927 15:20:46.274485 30970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37209 every 8 connection(s)
I20260927 15:20:46.275364 30903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:46.278013 30971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:46.278944 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 30903
I20260927 15:20:46.279075 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:46.279402 10186 external_mini_cluster.cc:1428] Setting key 8ffcdccd8b1864dd8d33a06982f2059c
I20260927 15:20:46.283737 30971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81: Bootstrap starting.
I20260927 15:20:46.285883 30971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:46.286737 30971 log.cc:824] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:46.288153 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:46.280734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58605 (local address 127.0.0.1:37209)
0927 15:20:46.281156 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:46.281170 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:46.281208 (+ 38us) server_negotiation.cc:407] Connection header received
0927 15:20:46.281864 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:46.281883 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:46.282217 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:46.282510 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:46.282919 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.283878 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.284735 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.285194 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.285332 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.286134 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:46.286141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:46.286165 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:46.286644 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:46.286707 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:46.286738 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:46.286847 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:46.287012 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 15:20:46.287228 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":134,"threads_started":1}
I20260927 15:20:46.288968 30971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81: No bootstrap required, opened a new log
I20260927 15:20:46.290983 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:46.292899 30971 raft_consensus.cc:348] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42bf320403b49c59e8ef09d02b0ca81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260927 15:20:46.293131 30971 raft_consensus.cc:374] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:46.293192 30971 raft_consensus.cc:729] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42bf320403b49c59e8ef09d02b0ca81, State: Initialized, Role: FOLLOWER
I20260927 15:20:46.293666 30971 consensus_queue.cc:260] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42bf320403b49c59e8ef09d02b0ca81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260927 15:20:46.293823 30971 raft_consensus.cc:388] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:46.293887 30971 raft_consensus.cc:482] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:46.293980 30971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:46.294962 30971 raft_consensus.cc:504] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42bf320403b49c59e8ef09d02b0ca81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260927 15:20:46.295342 30971 leader_election.cc:302] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b42bf320403b49c59e8ef09d02b0ca81; no voters:
I20260927 15:20:46.295671 30971 leader_election.cc:288] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:46.296063 30982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:46.296612 30982 raft_consensus.cc:686] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [term 1 LEADER]: Becoming Leader. State: Replica: b42bf320403b49c59e8ef09d02b0ca81, State: Running, Role: LEADER
I20260927 15:20:46.296940 30982 consensus_queue.cc:237] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42bf320403b49c59e8ef09d02b0ca81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } }
I20260927 15:20:46.297119 30974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:46.292860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56977 (local address 127.0.0.1:37209)
0927 15:20:46.292972 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:46.292977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:46.293171 (+ 194us) server_negotiation.cc:407] Connection header received
0927 15:20:46.293281 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:46.293284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:46.293330 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:46.293405 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:46.293692 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.294398 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.295272 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:46.295655 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:46.295730 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:46.296635 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:46.296639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:46.296642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:46.296655 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:46.296689 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:46.296703 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:46.296781 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:46.296901 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 15:20:46.296955 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:20:46.297017 30971 sys_catalog.cc:564] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:46.298933 30983 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b42bf320403b49c59e8ef09d02b0ca81. Latest consensus state: current_term: 1 leader_uuid: "b42bf320403b49c59e8ef09d02b0ca81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42bf320403b49c59e8ef09d02b0ca81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } } }
I20260927 15:20:46.298959 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b42bf320403b49c59e8ef09d02b0ca81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b42bf320403b49c59e8ef09d02b0ca81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37209 } } }
I20260927 15:20:46.299077 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:46.299571 30995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:46.299261 30983 sys_catalog.cc:458] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:46.302084 30995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:46.307322 30995 catalog_manager.cc:1347] Generated new cluster ID: 2d98984d37814d0281bd074e7682ace5
I20260927 15:20:46.307382 30995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:46.321055 30995 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:46.321978 30995 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:46.330462 30995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b42bf320403b49c59e8ef09d02b0ca81: Generated new TSK 0
I20260927 15:20:46.331377 30995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:46.340068 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 30903
2026-09-27T15:20:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:46 dist-test-slave-m4lt krb5kdc[31002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:46 dist-test-slave-m4lt krb5kdc[31002](info): set up 2 sockets
Sep 27 15:20:46 dist-test-slave-m4lt krb5kdc[31002](info): commencing operation
krb5kdc: starting...
W20260927 15:20:47.293779 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:47 dist-test-slave-m4lt krb5kdc[31002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:47.406364 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:47.516093 31015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:47.516345 31015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:47.516396 31015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:47.520359 31015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:47.520427 31015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:47.520444 31015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:47.520459 31015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:47.520478 31015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:47.525717 31015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:47.526744 31015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:47.527714 31015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:47.535776 31023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:47.535780 31020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:47.535770 31021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:47.536039 31015 server_base.cc:1034] running on GCE node
I20260927 15:20:47.536584 31015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:47.537678 31015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:47.538841 31015 hybrid_clock.cc:648] HybridClock initialized: now 1790522447538826 us; error 28 us; skew 500 ppm
Sep 27 15:20:47 dist-test-slave-m4lt krb5kdc[31002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:47.541348 31015 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:47.542217 31015 webserver.cc:460] Webserver started at http://127.0.0.1:35529/ using document root <none> and password file <none>
I20260927 15:20:47.542721 31015 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:47.542783 31015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:47.542968 31015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:47.544904 31015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "7b564bcc34094d07bf94f00f3122670a"
format_stamp: "Formatted at 2026-09-27 15:20:47 on dist-test-slave-m4lt"
server_key: "bfd9fdf42c3fed82432cc09e7b21e8bd"
server_key_iv: "c5e502c99bb7ba316ed07fb78706779a"
server_key_version: "encryptionkey@0"
I20260927 15:20:47.545414 31015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "7b564bcc34094d07bf94f00f3122670a"
format_stamp: "Formatted at 2026-09-27 15:20:47 on dist-test-slave-m4lt"
server_key: "bfd9fdf42c3fed82432cc09e7b21e8bd"
server_key_iv: "c5e502c99bb7ba316ed07fb78706779a"
server_key_version: "encryptionkey@0"
I20260927 15:20:47.548951 31015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:20:47.551226 31030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:47.552484 31015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 15:20:47.552591 31015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "7b564bcc34094d07bf94f00f3122670a"
format_stamp: "Formatted at 2026-09-27 15:20:47 on dist-test-slave-m4lt"
server_key: "bfd9fdf42c3fed82432cc09e7b21e8bd"
server_key_iv: "c5e502c99bb7ba316ed07fb78706779a"
server_key_version: "encryptionkey@0"
I20260927 15:20:47.552677 31015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:47.556939 31015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:47.559609 31015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:47.559760 31015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:47.566026 31015 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38085
I20260927 15:20:47.566056 31082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38085 every 8 connection(s)
I20260927 15:20:47.566943 31015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:47.569625 31083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:47.571915 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31015
I20260927 15:20:47.572016 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:47.572250 10186 external_mini_cluster.cc:1428] Setting key 95f3d7de0615c7a86906eab4510bc297
I20260927 15:20:47.575711 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a: Bootstrap starting.
I20260927 15:20:47.577757 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:47.573591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43513 (local address 127.0.0.1:38085)
0927 15:20:47.574047 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:47.574065 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:47.574108 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:20:47.574921 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:47.574944 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:47.575313 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:47.575678 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:47.576346 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:47.576351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:47.576366 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:47.576376 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:47.576434 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:47.576458 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:47.576585 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:47.576648 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 15:20:47.576817 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":121,"threads_started":1}
I20260927 15:20:47.578395 31083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:47.579257 31083 log.cc:824] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:47.580709 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:47.581569 31083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a: No bootstrap required, opened a new log
I20260927 15:20:47.584708 31086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:47.583097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57327 (local address 127.0.0.1:38085)
0927 15:20:47.583311 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:47.583315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:47.583412 (+ 97us) server_negotiation.cc:407] Connection header received
0927 15:20:47.583515 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:47.583517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:47.583550 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:47.583622 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:47.584334 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:47.584337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:47.584338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:47.584348 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:47.584372 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:47.584381 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:47.584448 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:47.584544 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 15:20:47.584581 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 15:20:47.586259 31083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b564bcc34094d07bf94f00f3122670a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260927 15:20:47.586500 31083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:47.586549 31083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b564bcc34094d07bf94f00f3122670a, State: Initialized, Role: FOLLOWER
I20260927 15:20:47.586990 31083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b564bcc34094d07bf94f00f3122670a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260927 15:20:47.587132 31083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:47.587199 31083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:47.587303 31083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:47.588258 31083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b564bcc34094d07bf94f00f3122670a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260927 15:20:47.588590 31083 leader_election.cc:302] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b564bcc34094d07bf94f00f3122670a; no voters:
I20260927 15:20:47.588953 31083 leader_election.cc:288] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:47.589107 31093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:47.589313 31093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [term 1 LEADER]: Becoming Leader. State: Replica: 7b564bcc34094d07bf94f00f3122670a, State: Running, Role: LEADER
I20260927 15:20:47.589643 31093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b564bcc34094d07bf94f00f3122670a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260927 15:20:47.590224 31083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:47.591249 31095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b564bcc34094d07bf94f00f3122670a. Latest consensus state: current_term: 1 leader_uuid: "7b564bcc34094d07bf94f00f3122670a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b564bcc34094d07bf94f00f3122670a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } } }
I20260927 15:20:47.591395 31095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:47.591665 31094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b564bcc34094d07bf94f00f3122670a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b564bcc34094d07bf94f00f3122670a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } } }
I20260927 15:20:47.591758 31102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:47.591778 31094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:47.594861 31102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:47.600368 31102 catalog_manager.cc:1347] Generated new cluster ID: 0c8e890cddee4839b9f7fc9a17866a5d
I20260927 15:20:47.600446 31102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:47.618271 31102 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:47.618660 31102 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:47.629329 31102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b564bcc34094d07bf94f00f3122670a: Generated new TSK 0
I20260927 15:20:47.630056 31102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:47.638005 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31015
2026-09-27T15:20:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:47 dist-test-slave-m4lt krb5kdc[31114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:47 dist-test-slave-m4lt krb5kdc[31114](info): set up 2 sockets
Sep 27 15:20:47 dist-test-slave-m4lt krb5kdc[31114](info): commencing operation
krb5kdc: starting...
W20260927 15:20:48.589483 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:48 dist-test-slave-m4lt krb5kdc[31114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:48.702497 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:48.809522 31127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:48.809809 31127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:48.809854 31127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:48.814113 31127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:48.814188 31127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:48.814215 31127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:48.814237 31127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:48.814258 31127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:48.819876 31127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:48.820859 31127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:48.821902 31127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:48.829952 31135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:48.829952 31133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:48.830071 31132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:48.830346 31127 server_base.cc:1034] running on GCE node
I20260927 15:20:48.830886 31127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:48.831974 31127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:48.833146 31127 hybrid_clock.cc:648] HybridClock initialized: now 1790522448833115 us; error 47 us; skew 500 ppm
Sep 27 15:20:48 dist-test-slave-m4lt krb5kdc[31114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:48.835881 31127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:48.836728 31127 webserver.cc:460] Webserver started at http://127.0.0.1:38849/ using document root <none> and password file <none>
I20260927 15:20:48.837217 31127 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:48.837272 31127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:48.837472 31127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:48.839444 31127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "6645ae47c0874c6fb3bf1342dab89544"
format_stamp: "Formatted at 2026-09-27 15:20:48 on dist-test-slave-m4lt"
server_key: "d5a3be8308efec956f25cf76f29c91fa"
server_key_iv: "1b1d511a4c97816ce8b91f22ee292998"
server_key_version: "encryptionkey@0"
I20260927 15:20:48.839924 31127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "6645ae47c0874c6fb3bf1342dab89544"
format_stamp: "Formatted at 2026-09-27 15:20:48 on dist-test-slave-m4lt"
server_key: "d5a3be8308efec956f25cf76f29c91fa"
server_key_iv: "1b1d511a4c97816ce8b91f22ee292998"
server_key_version: "encryptionkey@0"
I20260927 15:20:48.843454 31127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:20:48.845739 31142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:48.846951 31127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:48.847045 31127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "6645ae47c0874c6fb3bf1342dab89544"
format_stamp: "Formatted at 2026-09-27 15:20:48 on dist-test-slave-m4lt"
server_key: "d5a3be8308efec956f25cf76f29c91fa"
server_key_iv: "1b1d511a4c97816ce8b91f22ee292998"
server_key_version: "encryptionkey@0"
I20260927 15:20:48.847126 31127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:48.875998 31127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:48.879057 31127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:48.879220 31127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:48.886157 31127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42303
I20260927 15:20:48.886183 31194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42303 every 8 connection(s)
I20260927 15:20:48.887121 31127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:48.888247 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31127
I20260927 15:20:48.888365 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:48.888612 10186 external_mini_cluster.cc:1428] Setting key ff8994a922c5c6bf450fe55cd8b6bbd0
I20260927 15:20:48.890084 31195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:48.896689 31195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544: Bootstrap starting.
Sep 27 15:20:48 dist-test-slave-m4lt krb5kdc[31114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:20:48.899120 31195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:48.899978 31195 log.cc:824] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:48.901868 31195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544: No bootstrap required, opened a new log
I20260927 15:20:48.903091 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:48.890049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:53129 (local address 127.0.0.1:42303)
0927 15:20:48.890458 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:48.890476 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:48.890518 (+ 42us) server_negotiation.cc:407] Connection header received
0927 15:20:48.891333 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:48.891353 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:48.891769 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:48.892120 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:48.892750 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.894238 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.895109 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.895832 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.895982 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.898470 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:48.898477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:48.898492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:48.898522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:48.900345 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:48.900849 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:48.900855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:48.900859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:48.900963 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:48.901277 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:48.901279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:48.901280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:48.901723 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:48.901922 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:48.902210 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 15:20:48.902418 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":133,"threads_started":1}
I20260927 15:20:48.905409 31195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6645ae47c0874c6fb3bf1342dab89544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260927 15:20:48.905619 31195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:48.905704 31195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6645ae47c0874c6fb3bf1342dab89544, State: Initialized, Role: FOLLOWER
I20260927 15:20:48.905781 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:48.906117 31195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6645ae47c0874c6fb3bf1342dab89544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260927 15:20:48.906245 31195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:48.906316 31195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:48.906409 31195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:48.907281 31195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6645ae47c0874c6fb3bf1342dab89544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260927 15:20:48.907631 31195 leader_election.cc:302] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6645ae47c0874c6fb3bf1342dab89544; no voters:
I20260927 15:20:48.907948 31195 leader_election.cc:288] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:48.908030 31205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:48.908237 31205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [term 1 LEADER]: Becoming Leader. State: Replica: 6645ae47c0874c6fb3bf1342dab89544, State: Running, Role: LEADER
I20260927 15:20:48.908546 31205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6645ae47c0874c6fb3bf1342dab89544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } }
I20260927 15:20:48.909197 31195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:48.909678 31207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6645ae47c0874c6fb3bf1342dab89544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6645ae47c0874c6fb3bf1342dab89544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } } }
I20260927 15:20:48.909708 31205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6645ae47c0874c6fb3bf1342dab89544. Latest consensus state: current_term: 1 leader_uuid: "6645ae47c0874c6fb3bf1342dab89544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6645ae47c0874c6fb3bf1342dab89544" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42303 } } }
I20260927 15:20:48.909798 31207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:48.909806 31205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:48.910183 31210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:48.913376 31210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:48.915695 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:48.907639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:35473 (local address 127.0.0.1:42303)
0927 15:20:48.907737 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:48.907781 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:48.907989 (+ 208us) server_negotiation.cc:407] Connection header received
0927 15:20:48.908107 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:48.908110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:48.908152 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:48.908212 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:48.908833 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.909500 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.910610 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:48.910935 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:48.911413 (+ 478us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:48.912601 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:48.912605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:48.912608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:48.912632 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:48.914144 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:48.914571 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:48.914574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:48.914576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:48.914648 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:48.914940 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:48.914943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:48.914945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:48.915107 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:48.915229 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:48.915472 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 15:20:48.915553 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 15:20:48.922276 31210 catalog_manager.cc:1347] Generated new cluster ID: 8c9f7ec4ce9648ab8c3544e07ccaa675
I20260927 15:20:48.922338 31210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:48.961206 31210 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:48.962196 31210 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:48.967701 31210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6645ae47c0874c6fb3bf1342dab89544: Generated new TSK 0
I20260927 15:20:48.968376 31210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:48.977553 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31127
W20260927 15:20:48.981933 31112 connection.cc:535] client connection to 127.0.0.1:42303 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42303): Connection reset by peer (error 104)
2026-09-27T15:20:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:49 dist-test-slave-m4lt krb5kdc[31226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:49 dist-test-slave-m4lt krb5kdc[31226](info): set up 2 sockets
Sep 27 15:20:49 dist-test-slave-m4lt krb5kdc[31226](info): commencing operation
krb5kdc: starting...
W20260927 15:20:49.941679 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:50 dist-test-slave-m4lt krb5kdc[31226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:50.060055 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:50.166186 31239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:50.166477 31239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:50.166539 31239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:50.170537 31239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:50.170599 31239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:50.170624 31239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:50.170647 31239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:50.170674 31239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:50.175869 31239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:50.176877 31239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:50.177883 31239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:50.186133 31244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:50.186120 31247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:50.186187 31239 server_base.cc:1034] running on GCE node
W20260927 15:20:50.186120 31245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:50.186884 31239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:50.187932 31239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:50.189087 31239 hybrid_clock.cc:648] HybridClock initialized: now 1790522450189063 us; error 36 us; skew 500 ppm
Sep 27 15:20:50 dist-test-slave-m4lt krb5kdc[31226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:50.195606 31239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:50.196516 31239 webserver.cc:460] Webserver started at http://127.0.0.1:46729/ using document root <none> and password file <none>
I20260927 15:20:50.197014 31239 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:50.197069 31239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:50.197229 31239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:50.199261 31239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "6fddb6ae7e5b411a89bd4ad08d194478"
format_stamp: "Formatted at 2026-09-27 15:20:50 on dist-test-slave-m4lt"
server_key: "75160f87287da1992ebe4467de259051"
server_key_iv: "694de1eb11ea5af845b246be7170b6cb"
server_key_version: "encryptionkey@0"
I20260927 15:20:50.199765 31239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "6fddb6ae7e5b411a89bd4ad08d194478"
format_stamp: "Formatted at 2026-09-27 15:20:50 on dist-test-slave-m4lt"
server_key: "75160f87287da1992ebe4467de259051"
server_key_iv: "694de1eb11ea5af845b246be7170b6cb"
server_key_version: "encryptionkey@0"
I20260927 15:20:50.203336 31239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 15:20:50.205730 31254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:50.206976 31239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 15:20:50.207088 31239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "6fddb6ae7e5b411a89bd4ad08d194478"
format_stamp: "Formatted at 2026-09-27 15:20:50 on dist-test-slave-m4lt"
server_key: "75160f87287da1992ebe4467de259051"
server_key_iv: "694de1eb11ea5af845b246be7170b6cb"
server_key_version: "encryptionkey@0"
I20260927 15:20:50.207187 31239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:50.227160 31239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:50.230357 31239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:50.230530 31239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:50.237344 31239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34217
I20260927 15:20:50.237349 31306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34217 every 8 connection(s)
I20260927 15:20:50.238330 31239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:50.241036 31307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:50.246153 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31239
I20260927 15:20:50.246261 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:50.246417 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478: Bootstrap starting.
I20260927 15:20:50.246503 10186 external_mini_cluster.cc:1428] Setting key 5f3c25ad02578bb304946e4df40fba7b
I20260927 15:20:50.248737 31307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:50.249429 31307 log.cc:824] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:50.251400 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478: No bootstrap required, opened a new log
I20260927 15:20:50.255095 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:50.247769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:36093 (local address 127.0.0.1:34217)
0927 15:20:50.248174 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:50.248189 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:50.248223 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:20:50.248822 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:50.248840 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:50.249188 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:50.249464 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:50.249864 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.250921 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.252062 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.252517 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.252640 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.253446 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:50.253450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:50.253462 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:50.253865 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:50.253918 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:50.253941 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:50.254037 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:50.254212 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 15:20:50.254394 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260927 15:20:50.255177 31307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:20:50.255432 31307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:50.255494 31307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fddb6ae7e5b411a89bd4ad08d194478, State: Initialized, Role: FOLLOWER
I20260927 15:20:50.255903 31307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:20:50.256026 31307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:50.256088 31307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:50.256179 31307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:50.257097 31307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:20:50.257411 31307 leader_election.cc:302] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fddb6ae7e5b411a89bd4ad08d194478; no voters:
I20260927 15:20:50.257728 31307 leader_election.cc:288] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:50.257803 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:50.257810 31312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:50.258011 31312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [term 1 LEADER]: Becoming Leader. State: Replica: 6fddb6ae7e5b411a89bd4ad08d194478, State: Running, Role: LEADER
I20260927 15:20:50.258358 31312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } }
I20260927 15:20:50.258989 31307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:50.259728 31313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } } }
I20260927 15:20:50.259860 31313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:50.260164 31314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fddb6ae7e5b411a89bd4ad08d194478. Latest consensus state: current_term: 1 leader_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fddb6ae7e5b411a89bd4ad08d194478" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34217 } } }
I20260927 15:20:50.260245 31314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478 [sys.catalog]: This master's current role is: LEADER
W20260927 15:20:50.263762 31332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:50.263837 31332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:50.263893 31325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:50.264287 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:50.260225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:46709 (local address 127.0.0.1:34217)
0927 15:20:50.260332 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:50.260336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:50.260485 (+ 149us) server_negotiation.cc:407] Connection header received
0927 15:20:50.260636 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:50.260639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:50.260690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:50.260767 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:50.261160 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.261899 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.262747 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:50.262978 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:50.263062 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:50.263830 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:50.263834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:50.263836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:50.263849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:50.263876 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:50.263888 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:50.263958 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:50.264130 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 15:20:50.264162 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 15:20:50.264891 31325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:50.270105 31325 catalog_manager.cc:1347] Generated new cluster ID: 4fdb8ee49670487a8e543ec105288814
I20260927 15:20:50.270179 31325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:50.282603 31325 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:50.283516 31325 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:50.289773 31325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fddb6ae7e5b411a89bd4ad08d194478: Generated new TSK 0
I20260927 15:20:50.290566 31325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:50.301570 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31239
W20260927 15:20:50.305815 31224 connection.cc:535] client connection to 127.0.0.1:34217 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34217): Connection reset by peer (error 104)
2026-09-27T15:20:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:50 dist-test-slave-m4lt krb5kdc[31338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:50 dist-test-slave-m4lt krb5kdc[31338](info): set up 2 sockets
Sep 27 15:20:50 dist-test-slave-m4lt krb5kdc[31338](info): commencing operation
krb5kdc: starting...
W20260927 15:20:51.253552 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:51 dist-test-slave-m4lt krb5kdc[31338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:51.366864 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:51.472520 31351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:51.472824 31351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:51.472884 31351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:51.476979 31351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:51.477046 31351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:51.477073 31351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:51.477093 31351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:51.477114 31351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:51.482421 31351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:51.483453 31351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:51.484429 31351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:51.492358 31356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:51.492388 31359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:51.492401 31357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:51.492573 31351 server_base.cc:1034] running on GCE node
I20260927 15:20:51.493129 31351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:51.494223 31351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:51.495370 31351 hybrid_clock.cc:648] HybridClock initialized: now 1790522451495353 us; error 33 us; skew 500 ppm
Sep 27 15:20:51 dist-test-slave-m4lt krb5kdc[31338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:51.497934 31351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:51.498739 31351 webserver.cc:460] Webserver started at http://127.0.0.1:41439/ using document root <none> and password file <none>
I20260927 15:20:51.499236 31351 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:51.499341 31351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:51.499531 31351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:51.501492 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "c9c4ad41300a459eb67161f97de1668a"
format_stamp: "Formatted at 2026-09-27 15:20:51 on dist-test-slave-m4lt"
server_key: "65e0ad08201f4ca4564c34c44c88d297"
server_key_iv: "e07b0f43c43646735ef5ef9c7ee54d10"
server_key_version: "encryptionkey@0"
I20260927 15:20:51.502023 31351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "c9c4ad41300a459eb67161f97de1668a"
format_stamp: "Formatted at 2026-09-27 15:20:51 on dist-test-slave-m4lt"
server_key: "65e0ad08201f4ca4564c34c44c88d297"
server_key_iv: "e07b0f43c43646735ef5ef9c7ee54d10"
server_key_version: "encryptionkey@0"
I20260927 15:20:51.505488 31351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:20:51.507771 31366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:51.508970 31351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:51.509085 31351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "c9c4ad41300a459eb67161f97de1668a"
format_stamp: "Formatted at 2026-09-27 15:20:51 on dist-test-slave-m4lt"
server_key: "65e0ad08201f4ca4564c34c44c88d297"
server_key_iv: "e07b0f43c43646735ef5ef9c7ee54d10"
server_key_version: "encryptionkey@0"
I20260927 15:20:51.509173 31351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:51.513528 31351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:51.516337 31351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:51.516497 31351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:51.523008 31351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36611
I20260927 15:20:51.523053 31418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36611 every 8 connection(s)
I20260927 15:20:51.523929 31351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:51.526669 31419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:51.531860 31419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a: Bootstrap starting.
I20260927 15:20:51.532516 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31351
I20260927 15:20:51.532636 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:51.532892 10186 external_mini_cluster.cc:1428] Setting key 4fca87220a35668e7c661eee66a2f8bd
I20260927 15:20:51.534490 31419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:51.535197 31419 log.cc:824] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:51.537492 31419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a: No bootstrap required, opened a new log
I20260927 15:20:51.538056 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:51.534078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:48331 (local address 127.0.0.1:36611)
0927 15:20:51.534539 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:51.534554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:51.534591 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:20:51.535257 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:51.535275 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:51.535582 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:51.535914 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:51.536589 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:51.536594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:51.536609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:51.536619 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:51.536680 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:51.536707 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:51.536827 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:51.536923 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 15:20:51.537081 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260927 15:20:51.540795 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:51.542202 31419 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4ad41300a459eb67161f97de1668a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260927 15:20:51.542441 31419 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:51.542478 31419 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c4ad41300a459eb67161f97de1668a, State: Initialized, Role: FOLLOWER
I20260927 15:20:51.542898 31419 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4ad41300a459eb67161f97de1668a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260927 15:20:51.543031 31419 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:51.543112 31419 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:51.543231 31419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:51.544381 31419 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4ad41300a459eb67161f97de1668a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260927 15:20:51.544420 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:51.543026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:56825 (local address 127.0.0.1:36611)
0927 15:20:51.543152 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:51.543157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:51.543170 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:20:51.543258 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:51.543261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:51.543298 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:51.543369 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:51.544049 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:51.544052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:51.544053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:51.544063 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:51.544087 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:51.544098 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:51.544169 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:51.544272 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 15:20:51.544307 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 15:20:51.544735 31419 leader_election.cc:302] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9c4ad41300a459eb67161f97de1668a; no voters:
I20260927 15:20:51.545025 31419 leader_election.cc:288] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:51.545113 31429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:51.545364 31429 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [term 1 LEADER]: Becoming Leader. State: Replica: c9c4ad41300a459eb67161f97de1668a, State: Running, Role: LEADER
I20260927 15:20:51.545713 31429 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4ad41300a459eb67161f97de1668a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } }
I20260927 15:20:51.546265 31419 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:51.546830 31429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c4ad41300a459eb67161f97de1668a. Latest consensus state: current_term: 1 leader_uuid: "c9c4ad41300a459eb67161f97de1668a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4ad41300a459eb67161f97de1668a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } } }
I20260927 15:20:51.546938 31429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:51.547212 31431 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9c4ad41300a459eb67161f97de1668a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c4ad41300a459eb67161f97de1668a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36611 } } }
I20260927 15:20:51.547348 31431 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:51.547439 31436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:51.550472 31436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:51.556034 31436 catalog_manager.cc:1347] Generated new cluster ID: d3c6047867cf420d80d36ed39e84e32b
I20260927 15:20:51.556092 31436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:51.562897 31436 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:51.563169 31436 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:51.568837 31436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9c4ad41300a459eb67161f97de1668a: Generated new TSK 0
I20260927 15:20:51.569434 31436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:51.580202 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31351
2026-09-27T15:20:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:51 dist-test-slave-m4lt krb5kdc[31450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:51 dist-test-slave-m4lt krb5kdc[31450](info): set up 2 sockets
Sep 27 15:20:51 dist-test-slave-m4lt krb5kdc[31450](info): commencing operation
krb5kdc: starting...
W20260927 15:20:52.541814 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:52 dist-test-slave-m4lt krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:52.656946 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:52.763453 31463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:52.763698 31463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:52.763738 31463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:52.767660 31463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:52.767719 31463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:52.767736 31463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:52.767751 31463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:52.767764 31463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:52.772979 31463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:52.773983 31463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:52.774925 31463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:52.782912 31469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:52.782963 31468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:52.782917 31471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:52.782998 31463 server_base.cc:1034] running on GCE node
I20260927 15:20:52.783643 31463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:52.784648 31463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:52.785796 31463 hybrid_clock.cc:648] HybridClock initialized: now 1790522452785777 us; error 37 us; skew 500 ppm
Sep 27 15:20:52 dist-test-slave-m4lt krb5kdc[31450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:52.788352 31463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:52.789192 31463 webserver.cc:460] Webserver started at http://127.0.0.1:35699/ using document root <none> and password file <none>
I20260927 15:20:52.789754 31463 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:52.789821 31463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:52.790007 31463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:52.791937 31463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "144c7348cbd340f394f21b26f43fd448"
format_stamp: "Formatted at 2026-09-27 15:20:52 on dist-test-slave-m4lt"
server_key: "7bce023390fd67ef4d5892d9a297e1fc"
server_key_iv: "c466734be442b576dd6df82a4817b318"
server_key_version: "encryptionkey@0"
I20260927 15:20:52.792474 31463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "144c7348cbd340f394f21b26f43fd448"
format_stamp: "Formatted at 2026-09-27 15:20:52 on dist-test-slave-m4lt"
server_key: "7bce023390fd67ef4d5892d9a297e1fc"
server_key_iv: "c466734be442b576dd6df82a4817b318"
server_key_version: "encryptionkey@0"
I20260927 15:20:52.795981 31463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:20:52.798247 31478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:52.799508 31463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:52.799615 31463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "144c7348cbd340f394f21b26f43fd448"
format_stamp: "Formatted at 2026-09-27 15:20:52 on dist-test-slave-m4lt"
server_key: "7bce023390fd67ef4d5892d9a297e1fc"
server_key_iv: "c466734be442b576dd6df82a4817b318"
server_key_version: "encryptionkey@0"
I20260927 15:20:52.799705 31463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:52.815114 31463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:52.818187 31463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:52.818424 31463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:52.825438 31463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35113
I20260927 15:20:52.825467 31530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35113 every 8 connection(s)
I20260927 15:20:52.826509 31463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:52.829955 31531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:52.832930 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31463
I20260927 15:20:52.833084 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:52.833372 10186 external_mini_cluster.cc:1428] Setting key 51e42819bad74dc56772b8f388bdcbd6
I20260927 15:20:52.836233 31531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448: Bootstrap starting.
I20260927 15:20:52.838727 31531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:52.839632 31531 log.cc:824] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:52.841792 31531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448: No bootstrap required, opened a new log
Sep 27 15:20:52 dist-test-slave-m4lt krb5kdc[31450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 15:20:52.845295 31531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144c7348cbd340f394f21b26f43fd448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260927 15:20:52.845530 31531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:52.845584 31531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144c7348cbd340f394f21b26f43fd448, State: Initialized, Role: FOLLOWER
I20260927 15:20:52.846093 31531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144c7348cbd340f394f21b26f43fd448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260927 15:20:52.846235 31531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:52.846300 31531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:52.846407 31531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:52.847430 31531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144c7348cbd340f394f21b26f43fd448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260927 15:20:52.847747 31531 leader_election.cc:302] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 144c7348cbd340f394f21b26f43fd448; no voters:
I20260927 15:20:52.847924 31534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:52.834882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:39433 (local address 127.0.0.1:35113)
0927 15:20:52.835340 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:52.835355 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:52.835391 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:20:52.836035 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:52.836051 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:52.836420 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:52.836699 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:52.837603 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.838685 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.839875 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.840626 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.840753 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.843410 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:52.843415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:52.843428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:52.843456 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:52.844958 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:52.845456 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:52.845460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:52.845465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:52.845576 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:52.845958 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:52.845960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:52.845961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:52.846379 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:52.846582 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:52.846855 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:20:52.847072 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":155,"threads_started":1}
I20260927 15:20:52.848029 31531 leader_election.cc:288] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:52.848246 31536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:52.848465 31536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [term 1 LEADER]: Becoming Leader. State: Replica: 144c7348cbd340f394f21b26f43fd448, State: Running, Role: LEADER
I20260927 15:20:52.848812 31536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144c7348cbd340f394f21b26f43fd448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } }
I20260927 15:20:52.849303 31531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:52.850512 31537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "144c7348cbd340f394f21b26f43fd448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144c7348cbd340f394f21b26f43fd448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } } }
I20260927 15:20:52.850628 31537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:52.850883 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:52.850980 31538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 144c7348cbd340f394f21b26f43fd448. Latest consensus state: current_term: 1 leader_uuid: "144c7348cbd340f394f21b26f43fd448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "144c7348cbd340f394f21b26f43fd448" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35113 } } }
I20260927 15:20:52.851051 31538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448 [sys.catalog]: This master's current role is: LEADER
W20260927 15:20:52.854851 31551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:52.854928 31551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:52.854980 31546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:52.855885 31546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:52.861526 31534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:52.852997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:41421 (local address 127.0.0.1:35113)
0927 15:20:52.853186 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:52.853190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:52.853429 (+ 239us) server_negotiation.cc:407] Connection header received
0927 15:20:52.853590 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:52.853593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:52.853637 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:52.853708 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:52.854572 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.855286 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.856356 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:52.856668 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:52.856736 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:52.858338 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:52.858342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:52.858344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:52.858366 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:52.859611 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:52.860130 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:52.860134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:52.860135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:52.860219 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:52.860561 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:52.860565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:52.860566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:52.860768 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:52.860878 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:52.861286 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 15:20:52.861383 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 15:20:52.862155 31546 catalog_manager.cc:1347] Generated new cluster ID: 1c96abbccbdf4d008d6167942089d1ff
I20260927 15:20:52.862210 31546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:52.893392 31546 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:52.894548 31546 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:52.899669 31546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 144c7348cbd340f394f21b26f43fd448: Generated new TSK 0
I20260927 15:20:52.900328 31546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:52.908500 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31463
W20260927 15:20:52.912966 31448 connection.cc:535] client connection to 127.0.0.1:35113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35113): Connection reset by peer (error 104)
2026-09-27T15:20:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:52 dist-test-slave-m4lt krb5kdc[31562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:52 dist-test-slave-m4lt krb5kdc[31562](info): set up 2 sockets
Sep 27 15:20:52 dist-test-slave-m4lt krb5kdc[31562](info): commencing operation
krb5kdc: starting...
W20260927 15:20:53.857585 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:53 dist-test-slave-m4lt krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:53.970647 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:54.078666 31575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:54.078960 31575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:54.079020 31575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:54.082947 31575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:54.083003 31575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:54.083029 31575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:54.083048 31575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:54.083069 31575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:54.088249 31575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:54.089284 31575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:54.090267 31575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:54.098326 31580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:54.098331 31581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:54.098331 31583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:54.098380 31575 server_base.cc:1034] running on GCE node
I20260927 15:20:54.099071 31575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:54.100106 31575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:54.101264 31575 hybrid_clock.cc:648] HybridClock initialized: now 1790522454101240 us; error 32 us; skew 500 ppm
Sep 27 15:20:54 dist-test-slave-m4lt krb5kdc[31562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:54.103891 31575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:54.104719 31575 webserver.cc:460] Webserver started at http://127.0.0.1:37585/ using document root <none> and password file <none>
I20260927 15:20:54.105216 31575 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:54.105271 31575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:54.105432 31575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:54.107409 31575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "ad7c3a28523f408996e9b397dc29ce91"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-m4lt"
server_key: "8071ab5c359684993780045517aadd78"
server_key_iv: "000f7048cbc2f02003eb5837c7888203"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.107895 31575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "ad7c3a28523f408996e9b397dc29ce91"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-m4lt"
server_key: "8071ab5c359684993780045517aadd78"
server_key_iv: "000f7048cbc2f02003eb5837c7888203"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.111343 31575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 15:20:54.113631 31590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:54.114854 31575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:54.114959 31575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "ad7c3a28523f408996e9b397dc29ce91"
format_stamp: "Formatted at 2026-09-27 15:20:54 on dist-test-slave-m4lt"
server_key: "8071ab5c359684993780045517aadd78"
server_key_iv: "000f7048cbc2f02003eb5837c7888203"
server_key_version: "encryptionkey@0"
I20260927 15:20:54.115059 31575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:54.134218 31575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:54.137461 31575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:54.137645 31575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:54.144459 31575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39653
I20260927 15:20:54.144496 31642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39653 every 8 connection(s)
I20260927 15:20:54.145404 31575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:54.146554 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31575
I20260927 15:20:54.146690 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:54.146951 10186 external_mini_cluster.cc:1428] Setting key aa5b81761fbcaeb31daa2e7f3d80f752
I20260927 15:20:54.148350 31643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:54.153631 31643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91: Bootstrap starting.
I20260927 15:20:54.155597 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:54.148318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:55711 (local address 127.0.0.1:39653)
0927 15:20:54.148775 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:54.148791 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:54.148837 (+ 46us) server_negotiation.cc:407] Connection header received
0927 15:20:54.149485 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:54.149502 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:54.149930 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:54.150241 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:54.150629 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.151629 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.152514 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.152983 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.153130 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:54.154026 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:54.154030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:54.154043 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:54.154412 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:54.154462 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:54.154483 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:54.154594 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:54.154771 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 15:20:54.154911 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":158,"threads_started":1}
I20260927 15:20:54.155804 31643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:54.156471 31643 log.cc:824] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:54.158352 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:54.158488 31643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91: No bootstrap required, opened a new log
I20260927 15:20:54.161947 31643 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c3a28523f408996e9b397dc29ce91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260927 15:20:54.162159 31643 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:54.162238 31643 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7c3a28523f408996e9b397dc29ce91, State: Initialized, Role: FOLLOWER
I20260927 15:20:54.162647 31643 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c3a28523f408996e9b397dc29ce91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260927 15:20:54.162781 31643 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:54.162845 31643 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:54.162945 31643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:54.164049 31643 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c3a28523f408996e9b397dc29ce91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260927 15:20:54.164394 31643 leader_election.cc:302] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad7c3a28523f408996e9b397dc29ce91; no voters:
I20260927 15:20:54.164759 31643 leader_election.cc:288] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:54.165055 31653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:54.165305 31653 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [term 1 LEADER]: Becoming Leader. State: Replica: ad7c3a28523f408996e9b397dc29ce91, State: Running, Role: LEADER
I20260927 15:20:54.165835 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:54.160117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:46901 (local address 127.0.0.1:39653)
0927 15:20:54.160230 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:54.160234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:54.161909 (+ 1675us) server_negotiation.cc:407] Connection header received
0927 15:20:54.162024 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:54.162026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:54.162063 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:54.162120 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:54.162361 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.162882 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.164004 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:54.164236 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:54.164328 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:54.165246 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:54.165250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:54.165253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:54.165267 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:54.165296 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:54.165309 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:54.165447 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:54.165674 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 15:20:54.165714 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:20:54.165839 31653 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c3a28523f408996e9b397dc29ce91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } }
I20260927 15:20:54.166079 31643 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:54.167510 31655 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad7c3a28523f408996e9b397dc29ce91. Latest consensus state: current_term: 1 leader_uuid: "ad7c3a28523f408996e9b397dc29ce91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c3a28523f408996e9b397dc29ce91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } } }
I20260927 15:20:54.167559 31654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad7c3a28523f408996e9b397dc29ce91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad7c3a28523f408996e9b397dc29ce91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39653 } } }
I20260927 15:20:54.167701 31654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:54.167690 31655 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:54.168072 31666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:54.171003 31666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:54.176270 31666 catalog_manager.cc:1347] Generated new cluster ID: 6e8f5356a3804f928b9be791bb741ad2
I20260927 15:20:54.176386 31666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:54.188997 31666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:54.189983 31666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:54.202771 31666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad7c3a28523f408996e9b397dc29ce91: Generated new TSK 0
I20260927 15:20:54.203634 31666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:54.215986 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31575
W20260927 15:20:54.220526 31560 connection.cc:535] client connection to 127.0.0.1:39653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39653): Connection reset by peer (error 104)
2026-09-27T15:20:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:54 dist-test-slave-m4lt krb5kdc[31674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:54 dist-test-slave-m4lt krb5kdc[31674](info): set up 2 sockets
Sep 27 15:20:54 dist-test-slave-m4lt krb5kdc[31674](info): commencing operation
krb5kdc: starting...
W20260927 15:20:55.173622 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:55 dist-test-slave-m4lt krb5kdc[31674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 15:20:55.292325 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:55.400456 31687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:55.400745 31687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:55.400803 31687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:55.404932 31687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:55.405000 31687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:55.405026 31687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:55.405048 31687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:55.405073 31687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:55.410477 31687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:55.411530 31687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:55.412536 31687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:55.420797 31695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:55.420789 31693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:55.420789 31692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:55.421077 31687 server_base.cc:1034] running on GCE node
I20260927 15:20:55.421785 31687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:55.422880 31687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:55.424036 31687 hybrid_clock.cc:648] HybridClock initialized: now 1790522455424020 us; error 27 us; skew 500 ppm
Sep 27 15:20:55 dist-test-slave-m4lt krb5kdc[31674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:55.431815 31687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 15:20:55.432854 31687 webserver.cc:460] Webserver started at http://127.0.0.1:35381/ using document root <none> and password file <none>
I20260927 15:20:55.433434 31687 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:55.433499 31687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:55.433756 31687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:55.435858 31687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "555db7cba51149f191a3b01d5120e91e"
format_stamp: "Formatted at 2026-09-27 15:20:55 on dist-test-slave-m4lt"
server_key: "9a2a68e039e4522b520cf7d36beb5c27"
server_key_iv: "54342c0ed9ea5b4e6e729f1ebcfcd592"
server_key_version: "encryptionkey@0"
I20260927 15:20:55.436395 31687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "555db7cba51149f191a3b01d5120e91e"
format_stamp: "Formatted at 2026-09-27 15:20:55 on dist-test-slave-m4lt"
server_key: "9a2a68e039e4522b520cf7d36beb5c27"
server_key_iv: "54342c0ed9ea5b4e6e729f1ebcfcd592"
server_key_version: "encryptionkey@0"
I20260927 15:20:55.440065 31687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 15:20:55.442554 31702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:55.444017 31687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:20:55.444144 31687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "555db7cba51149f191a3b01d5120e91e"
format_stamp: "Formatted at 2026-09-27 15:20:55 on dist-test-slave-m4lt"
server_key: "9a2a68e039e4522b520cf7d36beb5c27"
server_key_iv: "54342c0ed9ea5b4e6e729f1ebcfcd592"
server_key_version: "encryptionkey@0"
I20260927 15:20:55.444249 31687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:55.448835 31687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:55.451895 31687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:55.452064 31687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:55.458510 31687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43063
I20260927 15:20:55.458525 31754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43063 every 8 connection(s)
I20260927 15:20:55.459407 31687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:55.462195 31755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:55.467811 31755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e: Bootstrap starting.
I20260927 15:20:55.468186 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31687
I20260927 15:20:55.468312 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:55.468580 10186 external_mini_cluster.cc:1428] Setting key b00042ca13ce78017826ddf941c1760d
I20260927 15:20:55.470450 31755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:55.471144 31755 log.cc:824] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:55.473093 31755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e: No bootstrap required, opened a new log
I20260927 15:20:55.473752 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:55.469916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:53615 (local address 127.0.0.1:43063)
0927 15:20:55.470363 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:55.470381 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:55.470418 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:20:55.471109 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:55.471127 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:55.471441 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:55.471750 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:55.472518 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:55.472522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:55.472535 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:55.472544 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:55.472596 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:55.472615 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:55.472707 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:55.472828 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 15:20:55.472997 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
I20260927 15:20:55.476513 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:55.477010 31755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555db7cba51149f191a3b01d5120e91e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43063 } }
I20260927 15:20:55.477227 31755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:55.477273 31755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555db7cba51149f191a3b01d5120e91e, State: Initialized, Role: FOLLOWER
I20260927 15:20:55.477741 31755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555db7cba51149f191a3b01d5120e91e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43063 } }
I20260927 15:20:55.477880 31755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:55.477941 31755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:55.478034 31755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:55.479126 31755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555db7cba51149f191a3b01d5120e91e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43063 } }
I20260927 15:20:55.479501 31755 leader_election.cc:302] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 555db7cba51149f191a3b01d5120e91e; no voters:
I20260927 15:20:55.479813 31755 leader_election.cc:288] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:55.479913 31765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:55.480039 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:55.478484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.245:49037 (local address 127.0.0.1:43063)
0927 15:20:55.478599 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:55.478604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:55.478774 (+ 170us) server_negotiation.cc:407] Connection header received
0927 15:20:55.478915 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:55.478918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:55.478951 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:55.479013 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:55.479668 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:55.479672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:55.479674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:55.479686 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:55.479723 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:55.479734 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:55.479831 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:55.479887 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 15:20:55.479927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:20:55.480427 31765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [term 1 LEADER]: Becoming Leader. State: Replica: 555db7cba51149f191a3b01d5120e91e, State: Running, Role: LEADER
I20260927 15:20:55.480732 31765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555db7cba51149f191a3b01d5120e91e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43063 } }
I20260927 15:20:55.481195 31755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:55.481961 31765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 555db7cba51149f191a3b01d5120e91e. Latest consensus state: current_term: 1 leader_uuid: "555db7cba51149f191a3b01d5120e91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555db7cba51149f191a3b01d5120e91e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43063 } } }
I20260927 15:20:55.482075 31765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:55.483080 31771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:55.486155 31771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:55.488083 31766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "555db7cba51149f191a3b01d5120e91e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555db7cba51149f191a3b01d5120e91e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43063 } } }
I20260927 15:20:55.488214 31766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:55.491775 31771 catalog_manager.cc:1347] Generated new cluster ID: 8f6903d141b644b8b58356776ba90b6a
I20260927 15:20:55.491829 31771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:55.503077 31771 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:55.503450 31771 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:55.511548 31771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 555db7cba51149f191a3b01d5120e91e: Generated new TSK 0
I20260927 15:20:55.512383 31771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:55.524545 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31687
2026-09-27T15:20:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:55 dist-test-slave-m4lt krb5kdc[31786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:55 dist-test-slave-m4lt krb5kdc[31786](info): set up 2 sockets
Sep 27 15:20:55 dist-test-slave-m4lt krb5kdc[31786](info): commencing operation
krb5kdc: starting...
W20260927 15:20:56.473579 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:56 dist-test-slave-m4lt krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:20:56.587975 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43023
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:43023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:56.696345 31799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:56.696611 31799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:56.696653 31799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:56.700640 31799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:56.700700 31799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:56.700718 31799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:56.700737 31799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:56.700764 31799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:56.706017 31799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:43023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:56.707021 31799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:56.708003 31799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:56.716137 31807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:56.716224 31804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:56.716199 31799 server_base.cc:1034] running on GCE node
W20260927 15:20:56.716151 31805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:56.716912 31799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:56.718017 31799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:56.719175 31799 hybrid_clock.cc:648] HybridClock initialized: now 1790522456719161 us; error 27 us; skew 500 ppm
Sep 27 15:20:56 dist-test-slave-m4lt krb5kdc[31786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522456, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:56.721808 31799 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:20:56.722680 31799 webserver.cc:460] Webserver started at http://127.9.242.190:40497/ using document root <none> and password file <none>
I20260927 15:20:56.723242 31799 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:56.723305 31799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:56.723522 31799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:56.725504 31799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "6f696b05a22e449595b329bfe0aebe96"
format_stamp: "Formatted at 2026-09-27 15:20:56 on dist-test-slave-m4lt"
server_key: "97c9efb73c8ab6ca7749b8e2e755a743"
server_key_iv: "6397b183bc878ebfa7d74b552ffdd087"
server_key_version: "encryptionkey@0"
I20260927 15:20:56.726085 31799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "6f696b05a22e449595b329bfe0aebe96"
format_stamp: "Formatted at 2026-09-27 15:20:56 on dist-test-slave-m4lt"
server_key: "97c9efb73c8ab6ca7749b8e2e755a743"
server_key_iv: "6397b183bc878ebfa7d74b552ffdd087"
server_key_version: "encryptionkey@0"
I20260927 15:20:56.729856 31799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:20:56.732263 31814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:56.733548 31799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:56.733708 31799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "6f696b05a22e449595b329bfe0aebe96"
format_stamp: "Formatted at 2026-09-27 15:20:56 on dist-test-slave-m4lt"
server_key: "97c9efb73c8ab6ca7749b8e2e755a743"
server_key_iv: "6397b183bc878ebfa7d74b552ffdd087"
server_key_version: "encryptionkey@0"
I20260927 15:20:56.733812 31799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:56.749826 31799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:56.752532 31799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:56.752701 31799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:56.759176 31799 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:43023
I20260927 15:20:56.759224 31866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:43023 every 8 connection(s)
I20260927 15:20:56.760121 31799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:56.762835 31867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:56.763605 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31799
I20260927 15:20:56.763716 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:56.763978 10186 external_mini_cluster.cc:1428] Setting key bde3c59d16a09ce05d6392c8cd7f8d69
I20260927 15:20:56.769163 31867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96: Bootstrap starting.
I20260927 15:20:56.771662 31867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:56.772336 31867 log.cc:824] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96: Log is configured to *not* fsync() on all Append() calls
Sep 27 15:20:56 dist-test-slave-m4lt krb5kdc[31786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:20:56.774361 31867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96: No bootstrap required, opened a new log
I20260927 15:20:56.778709 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:56.765291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39965 (local address 127.9.242.190:43023)
0927 15:20:56.765716 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:56.765736 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:56.765780 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:20:56.766568 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:56.766592 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:56.767033 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:56.767401 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:56.768168 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:56.769603 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:56.770492 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:56.771174 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:56.771320 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:56.773706 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:56.773712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:56.773728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:56.773765 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:56.775587 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:56.776086 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:56.776092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:56.776098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:56.776234 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:56.776561 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:56.776564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:56.776566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:56.777081 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:56.777274 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:56.777492 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 15:20:56.777739 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260927 15:20:56.778769 31867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f696b05a22e449595b329bfe0aebe96" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43023 } }
I20260927 15:20:56.779060 31867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:56.779114 31867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f696b05a22e449595b329bfe0aebe96, State: Initialized, Role: FOLLOWER
I20260927 15:20:56.779534 31867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f696b05a22e449595b329bfe0aebe96" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43023 } }
I20260927 15:20:56.779659 31867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:56.779721 31867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:56.779807 31867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:56.780918 31867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f696b05a22e449595b329bfe0aebe96" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43023 } }
I20260927 15:20:56.781266 31867 leader_election.cc:302] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f696b05a22e449595b329bfe0aebe96; no voters:
I20260927 15:20:56.781625 31867 leader_election.cc:288] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:56.781752 31872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:56.781831 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:56.781940 31872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [term 1 LEADER]: Becoming Leader. State: Replica: 6f696b05a22e449595b329bfe0aebe96, State: Running, Role: LEADER
I20260927 15:20:56.782241 31872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f696b05a22e449595b329bfe0aebe96" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43023 } }
I20260927 15:20:56.782927 31867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:56.783378 31872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f696b05a22e449595b329bfe0aebe96. Latest consensus state: current_term: 1 leader_uuid: "6f696b05a22e449595b329bfe0aebe96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f696b05a22e449595b329bfe0aebe96" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43023 } } }
I20260927 15:20:56.783499 31872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:56.783844 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f696b05a22e449595b329bfe0aebe96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f696b05a22e449595b329bfe0aebe96" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 43023 } } }
I20260927 15:20:56.783977 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96 [sys.catalog]: This master's current role is: LEADER
W20260927 15:20:56.787647 31891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:20:56.787714 31891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:20:56.787768 31881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:56.788393 31881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:56.794031 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:56.785349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51993 (local address 127.9.242.190:43023)
0927 15:20:56.785444 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:56.785449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:56.785911 (+ 462us) server_negotiation.cc:407] Connection header received
0927 15:20:56.786052 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:56.786055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:56.786097 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:56.786167 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:56.786924 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:56.787701 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:56.788837 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:56.789187 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:56.789287 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:56.790908 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:56.790912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:56.790914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:20:56.790935 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:56.792236 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:56.792662 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:56.792666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:56.792667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:56.792737 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:20:56.793109 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:20:56.793112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:20:56.793114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:20:56.793332 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:56.793483 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:56.793788 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 15:20:56.793883 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 15:20:56.795040 31881 catalog_manager.cc:1347] Generated new cluster ID: f224238234e3479fb4b4c338d979193b
I20260927 15:20:56.795099 31881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:56.815328 31881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:56.816409 31881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:56.821976 31881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f696b05a22e449595b329bfe0aebe96: Generated new TSK 0
I20260927 15:20:56.822705 31881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:56.827370 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31799
W20260927 15:20:56.831881 31784 connection.cc:535] client connection to 127.9.242.190:43023 recv error: Network error: failed to read from TLS socket (remote: 127.9.242.190:43023): Connection reset by peer (error 104)
2026-09-27T15:20:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:56 dist-test-slave-m4lt krb5kdc[31898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:56 dist-test-slave-m4lt krb5kdc[31898](info): set up 2 sockets
Sep 27 15:20:56 dist-test-slave-m4lt krb5kdc[31898](info): commencing operation
krb5kdc: starting...
W20260927 15:20:57.781687 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:57 dist-test-slave-m4lt krb5kdc[31898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:20:57.897425 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:44889
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:58.004618 31911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:58.004922 31911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:58.004982 31911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:58.009092 31911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:58.009172 31911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:58.009202 31911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:58.009223 31911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:58.009251 31911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:58.014751 31911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:44889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:58.015947 31911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:58.016994 31911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:58.025568 31917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:58.025568 31916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:58.025767 31911 server_base.cc:1034] running on GCE node
W20260927 15:20:58.025568 31919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:58.026424 31911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:58.027572 31911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:58.028733 31911 hybrid_clock.cc:648] HybridClock initialized: now 1790522458028711 us; error 37 us; skew 500 ppm
Sep 27 15:20:58 dist-test-slave-m4lt krb5kdc[31898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522458, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:58.031467 31911 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:20:58.032334 31911 webserver.cc:460] Webserver started at http://127.9.242.190:39735/ using document root <none> and password file <none>
I20260927 15:20:58.032871 31911 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:58.032935 31911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:58.033118 31911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:58.035173 31911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8"
format_stamp: "Formatted at 2026-09-27 15:20:58 on dist-test-slave-m4lt"
server_key: "42c2e4285016d9b7da1da363dcbe5332"
server_key_iv: "b19cb724f26c2d6391eb50c910cc7ffb"
server_key_version: "encryptionkey@0"
I20260927 15:20:58.035722 31911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8"
format_stamp: "Formatted at 2026-09-27 15:20:58 on dist-test-slave-m4lt"
server_key: "42c2e4285016d9b7da1da363dcbe5332"
server_key_iv: "b19cb724f26c2d6391eb50c910cc7ffb"
server_key_version: "encryptionkey@0"
I20260927 15:20:58.039462 31911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:20:58.041891 31926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:58.043184 31911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:20:58.043303 31911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8"
format_stamp: "Formatted at 2026-09-27 15:20:58 on dist-test-slave-m4lt"
server_key: "42c2e4285016d9b7da1da363dcbe5332"
server_key_iv: "b19cb724f26c2d6391eb50c910cc7ffb"
server_key_version: "encryptionkey@0"
I20260927 15:20:58.043434 31911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:58.053254 31911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:58.055936 31911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:58.056102 31911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:58.062610 31911 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:44889
I20260927 15:20:58.062623 31978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:44889 every 8 connection(s)
I20260927 15:20:58.063599 31911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:58.066596 31979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:58.072046 31979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8: Bootstrap starting.
I20260927 15:20:58.073087 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 31911
I20260927 15:20:58.073228 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:58.073503 10186 external_mini_cluster.cc:1428] Setting key 68e8ce027a3cf39df0378949f6947918
I20260927 15:20:58.074833 31979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:58.075536 31979 log.cc:824] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:58.077571 31979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8: No bootstrap required, opened a new log
I20260927 15:20:58.081872 31979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44889 } }
I20260927 15:20:58.082108 31979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:58.082175 31979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39868f2bf4404cfbb05a4ccd1aef7cc8, State: Initialized, Role: FOLLOWER
I20260927 15:20:58.082626 31979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44889 } }
I20260927 15:20:58.082724 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:58.074668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53087 (local address 127.9.242.190:44889)
0927 15:20:58.075057 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:58.075072 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:58.075106 (+ 34us) server_negotiation.cc:407] Connection header received
0927 15:20:58.075775 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:58.075794 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:58.076234 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:58.076536 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:58.076864 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:58.077880 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:58.079045 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:58.079557 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:58.079715 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:58.080812 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:58.080817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:58.080831 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:58.081235 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:58.081291 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:58.081313 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:58.081410 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:58.081597 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 15:20:58.081791 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
I20260927 15:20:58.082758 31979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:58.082883 31979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:58.082988 31979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:58.084164 31979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44889 } }
I20260927 15:20:58.084486 31979 leader_election.cc:302] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39868f2bf4404cfbb05a4ccd1aef7cc8; no voters:
I20260927 15:20:58.084812 31979 leader_election.cc:288] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:58.084921 31984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:58.085336 31984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [term 1 LEADER]: Becoming Leader. State: Replica: 39868f2bf4404cfbb05a4ccd1aef7cc8, State: Running, Role: LEADER
I20260927 15:20:58.085672 31984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44889 } }
I20260927 15:20:58.085832 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:58.086233 31979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:58.086731 31984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39868f2bf4404cfbb05a4ccd1aef7cc8. Latest consensus state: current_term: 1 leader_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44889 } } }
I20260927 15:20:58.086833 31984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:58.087136 31985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39868f2bf4404cfbb05a4ccd1aef7cc8" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 44889 } } }
I20260927 15:20:58.087195 31988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:58.087211 31985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:58.090175 31988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:58.093570 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:58.088215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33935 (local address 127.9.242.190:44889)
0927 15:20:58.088323 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:58.088327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:58.089382 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 15:20:58.089506 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:58.089509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:58.089557 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:58.089643 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:58.089924 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:58.090665 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:58.091971 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:20:58.092240 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:20:58.092301 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:20:58.093154 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:58.093158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:58.093161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:58.093173 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:58.093196 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:58.093206 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:58.093284 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:58.093409 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 15:20:58.093442 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 15:20:58.095562 31988 catalog_manager.cc:1347] Generated new cluster ID: 7aa2460376d14a5b9fb045820ad3ab6d
I20260927 15:20:58.095623 31988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:58.102182 31988 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:58.103094 31988 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:58.110385 31988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39868f2bf4404cfbb05a4ccd1aef7cc8: Generated new TSK 0
I20260927 15:20:58.111075 31988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:58.120644 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 31911
W20260927 15:20:58.124928 31896 connection.cc:535] client connection to 127.9.242.190:44889 recv error: Network error: failed to read from TLS socket (remote: 127.9.242.190:44889): Connection reset by peer (error 104)
2026-09-27T15:20:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:58 dist-test-slave-m4lt krb5kdc[32010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:58 dist-test-slave-m4lt krb5kdc[32010](info): set up 2 sockets
Sep 27 15:20:58 dist-test-slave-m4lt krb5kdc[32010](info): commencing operation
krb5kdc: starting...
W20260927 15:20:59.077631 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:20:59 dist-test-slave-m4lt krb5kdc[32010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:20:59.194330 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39557
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:39557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:20:59.301985 32023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:20:59.302244 32023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:20:59.302286 32023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:20:59.306591 32023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:20:59.306664 32023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:20:59.306684 32023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:20:59.306699 32023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:20:59.306713 32023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:20:59.312158 32023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:39557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:39557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:20:59.313158 32023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:20:59.314170 32023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:20:59.322338 32029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:59.322346 32028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:20:59.322356 32031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:20:59.322438 32023 server_base.cc:1034] running on GCE node
I20260927 15:20:59.323110 32023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:20:59.324213 32023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:20:59.325377 32023 hybrid_clock.cc:648] HybridClock initialized: now 1790522459325359 us; error 29 us; skew 500 ppm
Sep 27 15:20:59 dist-test-slave-m4lt krb5kdc[32010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522459, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:20:59.328033 32023 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:20:59.328938 32023 webserver.cc:460] Webserver started at http://127.9.242.190:44775/ using document root <none> and password file <none>
I20260927 15:20:59.329465 32023 fs_manager.cc:359] Metadata directory not provided
I20260927 15:20:59.329523 32023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:20:59.329735 32023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:20:59.331746 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "35ca4dd9aff84905931acf68466b84b6"
format_stamp: "Formatted at 2026-09-27 15:20:59 on dist-test-slave-m4lt"
server_key: "31ead111889359e3454cc25280c056ae"
server_key_iv: "a8c99c1078c78666e7ca8063530d9950"
server_key_version: "encryptionkey@0"
I20260927 15:20:59.332285 32023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "35ca4dd9aff84905931acf68466b84b6"
format_stamp: "Formatted at 2026-09-27 15:20:59 on dist-test-slave-m4lt"
server_key: "31ead111889359e3454cc25280c056ae"
server_key_iv: "a8c99c1078c78666e7ca8063530d9950"
server_key_version: "encryptionkey@0"
I20260927 15:20:59.336045 32023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:20:59.338469 32038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:20:59.339658 32023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:20:59.339761 32023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "35ca4dd9aff84905931acf68466b84b6"
format_stamp: "Formatted at 2026-09-27 15:20:59 on dist-test-slave-m4lt"
server_key: "31ead111889359e3454cc25280c056ae"
server_key_iv: "a8c99c1078c78666e7ca8063530d9950"
server_key_version: "encryptionkey@0"
I20260927 15:20:59.339864 32023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:20:59.344269 32023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:20:59.347172 32023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:20:59.347340 32023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:20:59.353526 32023 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:39557
I20260927 15:20:59.353565 32090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:39557 every 8 connection(s)
I20260927 15:20:59.354419 32023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:20:59.357009 32091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:20:59.359894 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 32023
I20260927 15:20:59.360033 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:20:59.360296 10186 external_mini_cluster.cc:1428] Setting key 1bc0fb3ba2b973c96f66e878aaea7c84
I20260927 15:20:59.362798 32091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6: Bootstrap starting.
I20260927 15:20:59.365217 32091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6: Neither blocks nor log segments found. Creating new log.
I20260927 15:20:59.365233 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:59.361565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46043 (local address 127.9.242.190:39557)
0927 15:20:59.361977 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:59.361992 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:59.362024 (+ 32us) server_negotiation.cc:407] Connection header received
0927 15:20:59.362622 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:59.362639 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:59.362980 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:59.363275 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:59.364047 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:59.364052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:59.364063 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:59.364072 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:59.364120 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:59.364138 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:59.364233 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:59.364342 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 15:20:59.364508 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":118,"threads_started":1}
I20260927 15:20:59.366003 32091 log.cc:824] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6: Log is configured to *not* fsync() on all Append() calls
I20260927 15:20:59.368003 32091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6: No bootstrap required, opened a new log
I20260927 15:20:59.368021 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:20:59.371909 32091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ca4dd9aff84905931acf68466b84b6" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39557 } }
I20260927 15:20:59.372125 32091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:20:59.372157 32091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ca4dd9aff84905931acf68466b84b6, State: Initialized, Role: FOLLOWER
I20260927 15:20:59.372572 32091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ca4dd9aff84905931acf68466b84b6" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39557 } }
I20260927 15:20:59.372733 32091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:20:59.372800 32091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:20:59.372900 32091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:20:59.373677 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:20:59.370107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32921 (local address 127.9.242.190:39557)
0927 15:20:59.370219 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 15:20:59.370223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:20:59.372496 (+ 2273us) server_negotiation.cc:407] Connection header received
0927 15:20:59.372539 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:20:59.372542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:20:59.372578 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:20:59.372664 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:20:59.373316 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:20:59.373320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:20:59.373323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:20:59.373335 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:20:59.373364 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:20:59.373377 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:20:59.373460 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:20:59.373502 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 15:20:59.373539 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:20:59.373968 32091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ca4dd9aff84905931acf68466b84b6" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39557 } }
I20260927 15:20:59.374301 32091 leader_election.cc:302] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35ca4dd9aff84905931acf68466b84b6; no voters:
I20260927 15:20:59.374624 32091 leader_election.cc:288] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:20:59.374696 32101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:20:59.374886 32101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [term 1 LEADER]: Becoming Leader. State: Replica: 35ca4dd9aff84905931acf68466b84b6, State: Running, Role: LEADER
I20260927 15:20:59.375223 32101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ca4dd9aff84905931acf68466b84b6" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39557 } }
I20260927 15:20:59.375885 32091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:20:59.376822 32103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35ca4dd9aff84905931acf68466b84b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ca4dd9aff84905931acf68466b84b6" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39557 } } }
I20260927 15:20:59.376938 32103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:59.377295 32110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:20:59.377321 32102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35ca4dd9aff84905931acf68466b84b6. Latest consensus state: current_term: 1 leader_uuid: "35ca4dd9aff84905931acf68466b84b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ca4dd9aff84905931acf68466b84b6" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 39557 } } }
I20260927 15:20:59.377465 32102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6 [sys.catalog]: This master's current role is: LEADER
I20260927 15:20:59.380125 32110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:20:59.385433 32110 catalog_manager.cc:1347] Generated new cluster ID: 57a2610c8abd41d083e249510c369b0b
I20260927 15:20:59.385509 32110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:20:59.402242 32110 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:20:59.402644 32110 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:20:59.409171 32110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35ca4dd9aff84905931acf68466b84b6: Generated new TSK 0
I20260927 15:20:59.409914 32110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:20:59.413767 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 32023
2026-09-27T15:20:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:20:59 dist-test-slave-m4lt krb5kdc[32122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:20:59 dist-test-slave-m4lt krb5kdc[32122](info): set up 2 sockets
Sep 27 15:20:59 dist-test-slave-m4lt krb5kdc[32122](info): commencing operation
krb5kdc: starting...
W20260927 15:21:00.361794 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:21:00 dist-test-slave-m4lt krb5kdc[32122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:21:00.476647 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36355
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:00.585202 32135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:00.585543 32135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:00.585590 32135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:00.589725 32135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:00.589784 32135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:00.589802 32135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:00.589818 32135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:00.589833 32135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:00.595757 32135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:36355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:36355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:00.596850 32135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:00.597936 32135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:00.606076 32143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:00.606056 32140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:00.606079 32141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:00.606419 32135 server_base.cc:1034] running on GCE node
I20260927 15:21:00.606971 32135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:00.608040 32135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:00.609184 32135 hybrid_clock.cc:648] HybridClock initialized: now 1790522460609162 us; error 42 us; skew 500 ppm
Sep 27 15:21:00 dist-test-slave-m4lt krb5kdc[32122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:00.615736 32135 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:21:00.616703 32135 webserver.cc:460] Webserver started at http://127.9.242.190:34591/ using document root <none> and password file <none>
I20260927 15:21:00.617246 32135 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:00.617308 32135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:00.617494 32135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:00.619571 32135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "75d34c4e65aa4bb6a30fe8528d597760"
format_stamp: "Formatted at 2026-09-27 15:21:00 on dist-test-slave-m4lt"
server_key: "bfefd60dd23718de15f83b6484428165"
server_key_iv: "f998ee5246e6b85f479720b74c0de26c"
server_key_version: "encryptionkey@0"
I20260927 15:21:00.620110 32135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "75d34c4e65aa4bb6a30fe8528d597760"
format_stamp: "Formatted at 2026-09-27 15:21:00 on dist-test-slave-m4lt"
server_key: "bfefd60dd23718de15f83b6484428165"
server_key_iv: "f998ee5246e6b85f479720b74c0de26c"
server_key_version: "encryptionkey@0"
I20260927 15:21:00.623996 32135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:21:00.626490 32150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:00.627730 32135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:21:00.627856 32135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "75d34c4e65aa4bb6a30fe8528d597760"
format_stamp: "Formatted at 2026-09-27 15:21:00 on dist-test-slave-m4lt"
server_key: "bfefd60dd23718de15f83b6484428165"
server_key_iv: "f998ee5246e6b85f479720b74c0de26c"
server_key_version: "encryptionkey@0"
I20260927 15:21:00.627954 32135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:00.640415 32135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:00.643282 32135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:00.643457 32135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:00.650575 32135 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:36355
I20260927 15:21:00.650594 32202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:36355 every 8 connection(s)
I20260927 15:21:00.651590 32135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:21:00.652473 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 32135
I20260927 15:21:00.652590 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:21:00.652868 10186 external_mini_cluster.cc:1428] Setting key 95c5fc27f81d32f43fd2114eae68ab4f
I20260927 15:21:00.654583 32203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:00.661207 32203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760: Bootstrap starting.
Sep 27 15:21:00 dist-test-slave-m4lt krb5kdc[32122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:00.663707 32203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:00.664588 32203 log.cc:824] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:00.666906 32203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760: No bootstrap required, opened a new log
I20260927 15:21:00.667672 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:00.654362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52399 (local address 127.9.242.190:36355)
0927 15:21:00.654766 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:00.654784 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:00.654824 (+ 40us) server_negotiation.cc:407] Connection header received
0927 15:21:00.655650 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:00.655671 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:00.656093 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:00.656463 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:00.657142 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:00.658659 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:00.659487 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:00.660177 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:00.660342 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:00.662723 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:00.662730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:00.662745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:00.662778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:00.664599 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:00.665027 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:00.665032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:00.665037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:00.665144 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:00.665430 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:00.665433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:00.665435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:00.665958 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:00.666149 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:00.666422 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 15:21:00.666717 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260927 15:21:00.670463 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:21:00.670727 32203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d34c4e65aa4bb6a30fe8528d597760" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36355 } }
I20260927 15:21:00.670953 32203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:00.671003 32203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75d34c4e65aa4bb6a30fe8528d597760, State: Initialized, Role: FOLLOWER
I20260927 15:21:00.671423 32203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d34c4e65aa4bb6a30fe8528d597760" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36355 } }
I20260927 15:21:00.671545 32203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:21:00.671653 32203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:21:00.671748 32203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:00.672801 32203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d34c4e65aa4bb6a30fe8528d597760" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36355 } }
I20260927 15:21:00.673166 32203 leader_election.cc:302] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75d34c4e65aa4bb6a30fe8528d597760; no voters:
I20260927 15:21:00.673470 32203 leader_election.cc:288] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:21:00.673542 32213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:00.673763 32213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [term 1 LEADER]: Becoming Leader. State: Replica: 75d34c4e65aa4bb6a30fe8528d597760, State: Running, Role: LEADER
I20260927 15:21:00.674115 32213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d34c4e65aa4bb6a30fe8528d597760" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36355 } }
I20260927 15:21:00.674842 32203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:00.675952 32215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75d34c4e65aa4bb6a30fe8528d597760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d34c4e65aa4bb6a30fe8528d597760" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36355 } } }
I20260927 15:21:00.676015 32214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75d34c4e65aa4bb6a30fe8528d597760. Latest consensus state: current_term: 1 leader_uuid: "75d34c4e65aa4bb6a30fe8528d597760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75d34c4e65aa4bb6a30fe8528d597760" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 36355 } } }
I20260927 15:21:00.676126 32214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:00.676081 32215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:00.677420 32222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:21:00.680567 32222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:21:00.680688 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:00.672462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40065 (local address 127.9.242.190:36355)
0927 15:21:00.672582 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:00.672586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:00.672695 (+ 109us) server_negotiation.cc:407] Connection header received
0927 15:21:00.672799 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:00.672801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:00.672850 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:00.672921 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:00.673647 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:00.674393 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:00.675280 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:00.675624 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:00.675703 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:00.677395 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:00.677399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:00.677401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:00.677426 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:00.678968 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:00.679422 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:00.679425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:00.679427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:00.679511 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:00.679777 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:00.679781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:00.679783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:00.679997 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:00.680124 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:00.680368 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 15:21:00.680484 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:21:00.686394 32222 catalog_manager.cc:1347] Generated new cluster ID: d99426c70b2d4628b47c2ccae5fa4186
I20260927 15:21:00.686462 32222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:21:00.705529 32222 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:21:00.706843 32222 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:21:00.714684 32222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75d34c4e65aa4bb6a30fe8528d597760: Generated new TSK 0
I20260927 15:21:00.715549 32222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:21:00.722549 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 32135
2026-09-27T15:21:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:21:00 dist-test-slave-m4lt krb5kdc[32234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:21:00 dist-test-slave-m4lt krb5kdc[32234](info): set up 2 sockets
Sep 27 15:21:00 dist-test-slave-m4lt krb5kdc[32234](info): commencing operation
krb5kdc: starting...
W20260927 15:21:01.673911 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:21:01 dist-test-slave-m4lt krb5kdc[32234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:21:01.790191 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45273
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:38185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:01.901769 32247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:01.902037 32247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:01.902084 32247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:01.906299 32247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:01.906359 32247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:01.906378 32247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:01.906395 32247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:01.906422 32247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:01.911705 32247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:38185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:01.912740 32247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:01.913794 32247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:01.925400 32255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:01.925400 32252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:01.925851 32253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:01.926482 32247 server_base.cc:1034] running on GCE node
I20260927 15:21:01.927184 32247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:01.928818 32247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:01.929960 32247 hybrid_clock.cc:648] HybridClock initialized: now 1790522461929929 us; error 49 us; skew 500 ppm
Sep 27 15:21:01 dist-test-slave-m4lt krb5kdc[32234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522461, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:01.933352 32247 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:21:01.934510 32247 webserver.cc:460] Webserver started at http://127.9.242.190:37693/ using document root <none> and password file <none>
I20260927 15:21:01.935237 32247 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:01.935362 32247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:01.935606 32247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:01.938638 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "7c145aad60a1441285279c70bd93fb6b"
format_stamp: "Formatted at 2026-09-27 15:21:01 on dist-test-slave-m4lt"
server_key: "4fc4ba3b275af3e6ce1ebd87c04684dd"
server_key_iv: "29289894fe2dbfab66f704f596e9d51c"
server_key_version: "encryptionkey@0"
I20260927 15:21:01.939368 32247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "7c145aad60a1441285279c70bd93fb6b"
format_stamp: "Formatted at 2026-09-27 15:21:01 on dist-test-slave-m4lt"
server_key: "4fc4ba3b275af3e6ce1ebd87c04684dd"
server_key_iv: "29289894fe2dbfab66f704f596e9d51c"
server_key_version: "encryptionkey@0"
I20260927 15:21:01.944170 32247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 15:21:01.947135 32262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:01.948766 32247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 15:21:01.948907 32247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "7c145aad60a1441285279c70bd93fb6b"
format_stamp: "Formatted at 2026-09-27 15:21:01 on dist-test-slave-m4lt"
server_key: "4fc4ba3b275af3e6ce1ebd87c04684dd"
server_key_iv: "29289894fe2dbfab66f704f596e9d51c"
server_key_version: "encryptionkey@0"
I20260927 15:21:01.949034 32247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:01.978842 32247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:01.982040 32247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:01.982209 32247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:01.989217 32247 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:45273
I20260927 15:21:01.989323 32314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:45273 every 8 connection(s)
I20260927 15:21:01.990299 32247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:21:01.991662 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 32247
I20260927 15:21:01.991791 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:21:01.992094 10186 external_mini_cluster.cc:1428] Setting key 65ee90110d70d9cce43497adea6caef7
I20260927 15:21:01.993610 32315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:02.000304 32315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b: Bootstrap starting.
I20260927 15:21:02.001508 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:01.993500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33031 (local address 127.9.242.190:45273)
0927 15:21:01.993961 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:01.993976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:01.994013 (+ 37us) server_negotiation.cc:407] Connection header received
0927 15:21:01.994687 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:01.994706 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:01.995063 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:01.995414 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:01.995758 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:01.996938 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:01.998220 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:01.998720 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:01.998874 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:01.999731 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:01.999736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:01.999750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:21:02.000145 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:02.000202 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:21:02.000225 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:02.000348 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.000528 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.000757 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":176,"threads_started":1}
I20260927 15:21:02.002696 32315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:02.003444 32315 log.cc:824] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:02.004380 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:21:02.005690 32315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b: No bootstrap required, opened a new log
I20260927 15:21:02.010000 32315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c145aad60a1441285279c70bd93fb6b" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45273 } }
I20260927 15:21:02.010334 32315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:02.010411 32315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c145aad60a1441285279c70bd93fb6b, State: Initialized, Role: FOLLOWER
I20260927 15:21:02.010805 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:02.006618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49247 (local address 127.9.242.190:45273)
0927 15:21:02.006736 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:02.006740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:02.006924 (+ 184us) server_negotiation.cc:407] Connection header received
0927 15:21:02.007037 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:02.007040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:02.007079 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:02.007139 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:02.007413 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.007922 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.009119 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:02.009394 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:02.009475 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:02.010355 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:02.010360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:02.010363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:21:02.010378 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:02.010411 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:21:02.010426 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:02.010507 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:02.010605 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 15:21:02.010664 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 15:21:02.010931 32315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c145aad60a1441285279c70bd93fb6b" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45273 } }
I20260927 15:21:02.011098 32315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:21:02.011168 32315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:21:02.011276 32315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:02.012233 32315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c145aad60a1441285279c70bd93fb6b" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45273 } }
I20260927 15:21:02.012574 32315 leader_election.cc:302] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c145aad60a1441285279c70bd93fb6b; no voters:
I20260927 15:21:02.012882 32315 leader_election.cc:288] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:21:02.013000 32325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:02.013223 32325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [term 1 LEADER]: Becoming Leader. State: Replica: 7c145aad60a1441285279c70bd93fb6b, State: Running, Role: LEADER
I20260927 15:21:02.013543 32325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c145aad60a1441285279c70bd93fb6b" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45273 } }
I20260927 15:21:02.014212 32315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:02.015132 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c145aad60a1441285279c70bd93fb6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c145aad60a1441285279c70bd93fb6b" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45273 } } }
I20260927 15:21:02.015270 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:02.015571 32327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c145aad60a1441285279c70bd93fb6b. Latest consensus state: current_term: 1 leader_uuid: "7c145aad60a1441285279c70bd93fb6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c145aad60a1441285279c70bd93fb6b" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45273 } } }
I20260927 15:21:02.015666 32327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:02.015635 32334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:21:02.018565 32334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:21:02.024457 32334 catalog_manager.cc:1347] Generated new cluster ID: 89bc918f75b946bb94588dacc99ed120
I20260927 15:21:02.024535 32334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:21:02.038965 32334 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:21:02.039979 32334 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:21:02.044889 32334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c145aad60a1441285279c70bd93fb6b: Generated new TSK 0
I20260927 15:21:02.045570 32334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:21:02.054067 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 32247
2026-09-27T15:21:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:21:02 dist-test-slave-m4lt krb5kdc[32346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:21:02 dist-test-slave-m4lt krb5kdc[32346](info): set up 2 sockets
Sep 27 15:21:02 dist-test-slave-m4lt krb5kdc[32346](info): commencing operation
krb5kdc: starting...
W20260927 15:21:03.005872 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:21:03 dist-test-slave-m4lt krb5kdc[32346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:21:03.120491 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45695
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:45695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:03.226461 32359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:03.226719 32359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:03.226760 32359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:03.230795 32359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:03.230865 32359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:03.230882 32359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:03.230897 32359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:03.230911 32359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:03.236147 32359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:45695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:45695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:03.237141 32359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:03.238127 32359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:03.246207 32367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:03.246423 32359 server_base.cc:1034] running on GCE node
W20260927 15:21:03.246222 32364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:03.246272 32365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:03.247186 32359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:03.248315 32359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:03.249487 32359 hybrid_clock.cc:648] HybridClock initialized: now 1790522463249460 us; error 38 us; skew 500 ppm
Sep 27 15:21:03 dist-test-slave-m4lt krb5kdc[32346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522463, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:03.252311 32359 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:21:03.253214 32359 webserver.cc:460] Webserver started at http://127.9.242.190:44515/ using document root <none> and password file <none>
I20260927 15:21:03.253777 32359 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:03.253840 32359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:03.254020 32359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:03.256021 32359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "9e7bc6c57e84420bbd6b9e72651555c7"
format_stamp: "Formatted at 2026-09-27 15:21:03 on dist-test-slave-m4lt"
server_key: "26badedc6b41ae33ef9e496541b2c177"
server_key_iv: "3846b3baca4aeee603f856c042f43f78"
server_key_version: "encryptionkey@0"
I20260927 15:21:03.256560 32359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "9e7bc6c57e84420bbd6b9e72651555c7"
format_stamp: "Formatted at 2026-09-27 15:21:03 on dist-test-slave-m4lt"
server_key: "26badedc6b41ae33ef9e496541b2c177"
server_key_iv: "3846b3baca4aeee603f856c042f43f78"
server_key_version: "encryptionkey@0"
I20260927 15:21:03.260244 32359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 15:21:03.262549 32374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:03.263773 32359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:21:03.263890 32359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "9e7bc6c57e84420bbd6b9e72651555c7"
format_stamp: "Formatted at 2026-09-27 15:21:03 on dist-test-slave-m4lt"
server_key: "26badedc6b41ae33ef9e496541b2c177"
server_key_iv: "3846b3baca4aeee603f856c042f43f78"
server_key_version: "encryptionkey@0"
I20260927 15:21:03.263979 32359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:03.268333 32359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:03.271291 32359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:03.271466 32359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:03.278015 32359 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:45695
I20260927 15:21:03.278025 32426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:45695 every 8 connection(s)
I20260927 15:21:03.278920 32359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:21:03.281697 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:03.286073 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 32359
I20260927 15:21:03.286238 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:21:03.286469 10186 external_mini_cluster.cc:1428] Setting key 0c90f4f6416b8419c5b4634f6b98eb5d
I20260927 15:21:03.287628 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7: Bootstrap starting.
I20260927 15:21:03.290292 32427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:03.291184 32427 log.cc:824] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:03.292194 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:03.287936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46619 (local address 127.9.242.190:45695)
0927 15:21:03.288365 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:03.288383 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:03.288424 (+ 41us) server_negotiation.cc:407] Connection header received
0927 15:21:03.289272 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:03.289296 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:03.289687 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:03.290075 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:03.290845 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:03.290850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:03.290864 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:21:03.290875 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:03.290933 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:21:03.290960 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:03.291067 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:03.291230 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 15:21:03.291382 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
I20260927 15:21:03.293187 32427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7: No bootstrap required, opened a new log
I20260927 15:21:03.295188 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:21:03.297117 32427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45695 } }
I20260927 15:21:03.297364 32427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 15:21:03.297422 32427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7bc6c57e84420bbd6b9e72651555c7, State: Initialized, Role: FOLLOWER
I20260927 15:21:03.297840 32427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45695 } }
I20260927 15:21:03.297974 32427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 15:21:03.298036 32427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 15:21:03.298144 32427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:03.298682 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:03.297049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52497 (local address 127.9.242.190:45695)
0927 15:21:03.297144 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:03.297147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:03.297352 (+ 205us) server_negotiation.cc:407] Connection header received
0927 15:21:03.297482 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:03.297486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:03.297520 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:03.297558 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:03.298349 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:03.298351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:03.298354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 15:21:03.298365 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:03.298392 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 15:21:03.298404 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:03.298489 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:03.298534 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 15:21:03.298567 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 15:21:03.299194 32427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45695 } }
I20260927 15:21:03.299528 32427 leader_election.cc:302] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e7bc6c57e84420bbd6b9e72651555c7; no voters:
I20260927 15:21:03.299798 32427 leader_election.cc:288] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 15:21:03.299888 32437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:03.300065 32437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [term 1 LEADER]: Becoming Leader. State: Replica: 9e7bc6c57e84420bbd6b9e72651555c7, State: Running, Role: LEADER
I20260927 15:21:03.300385 32437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45695 } }
I20260927 15:21:03.300892 32427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:03.301862 32439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e7bc6c57e84420bbd6b9e72651555c7. Latest consensus state: current_term: 1 leader_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45695 } } }
I20260927 15:21:03.301982 32439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:03.302282 32438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bc6c57e84420bbd6b9e72651555c7" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 45695 } } }
I20260927 15:21:03.302413 32438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:03.302336 32446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:21:03.305119 32446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:21:03.309818 32446 catalog_manager.cc:1347] Generated new cluster ID: 98a08793b5b4400ea135ee358cf2c61a
I20260927 15:21:03.309888 32446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:21:03.343268 32446 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:21:03.343628 32446 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:21:03.361107 32446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e7bc6c57e84420bbd6b9e72651555c7: Generated new TSK 0
I20260927 15:21:03.361899 32446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 15:21:03.366752 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 32359
2026-09-27T15:21:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1312 ms)
[----------] 18 tests from AuthTokenIssuingTest (23590 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:21:03 dist-test-slave-m4lt krb5kdc[32458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:21:03 dist-test-slave-m4lt krb5kdc[32458](info): set up 2 sockets
Sep 27 15:21:03 dist-test-slave-m4lt krb5kdc[32458](info): commencing operation
krb5kdc: starting...
W20260927 15:21:04.321698 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:21:04.441354 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:35631
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:35631,127.9.242.189:37693,127.9.242.188:44845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:04.549515 32471 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:21:04.549861 32471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:04.549908 32471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:04.549964 32471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:04.553897 32471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:04.553959 32471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:04.553985 32471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:04.554006 32471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:04.554028 32471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:04.559295 32471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:35631,127.9.242.189:37693,127.9.242.188:44845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:04.560345 32471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:04.561331 32471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:04.569402 32476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:04.569461 32479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:04.569413 32477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:04.569703 32471 server_base.cc:1034] running on GCE node
I20260927 15:21:04.570303 32471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:04.571419 32471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:04.572593 32471 hybrid_clock.cc:648] HybridClock initialized: now 1790522464572566 us; error 42 us; skew 500 ppm
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:04.575460 32471 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:21:04.576400 32471 webserver.cc:460] Webserver started at http://127.9.242.190:38153/ using document root <none> and password file <none>
I20260927 15:21:04.576937 32471 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:04.577004 32471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:04.577210 32471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:04.579314 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "c8f67288af1943308a6f7daf1fcd23fb"
format_stamp: "Formatted at 2026-09-27 15:21:04 on dist-test-slave-m4lt"
server_key: "6310409b857e392e6939fd52e7aa37b2"
server_key_iv: "846a2bf098d23c77ea03847e34cf55a5"
server_key_version: "encryptionkey@0"
I20260927 15:21:04.579859 32471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "c8f67288af1943308a6f7daf1fcd23fb"
format_stamp: "Formatted at 2026-09-27 15:21:04 on dist-test-slave-m4lt"
server_key: "6310409b857e392e6939fd52e7aa37b2"
server_key_iv: "846a2bf098d23c77ea03847e34cf55a5"
server_key_version: "encryptionkey@0"
I20260927 15:21:04.583741 32471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 15:21:04.586263 32486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:04.587507 32471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:21:04.587635 32471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "c8f67288af1943308a6f7daf1fcd23fb"
format_stamp: "Formatted at 2026-09-27 15:21:04 on dist-test-slave-m4lt"
server_key: "6310409b857e392e6939fd52e7aa37b2"
server_key_iv: "846a2bf098d23c77ea03847e34cf55a5"
server_key_version: "encryptionkey@0"
I20260927 15:21:04.587746 32471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:04.611805 32471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:04.614908 32471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:04.615087 32471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:04.621513 32471 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:35631
I20260927 15:21:04.621806 32538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:35631 every 8 connection(s)
I20260927 15:21:04.622486 32471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:21:04.625159 32539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:04.627701 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 32471
I20260927 15:21:04.627856 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:21:04.628132 10186 external_mini_cluster.cc:1428] Setting key 493a6ab1af5413044313d778cd801d98
I20260927 15:21:04.629222 32539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.242.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.189@KRBTEST.COM" created.
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:04.644713 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.631729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35631 (local address 127.0.0.1:40274)
0927 15:21:04.632266 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.632308 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:04.633264 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:04.633933 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:04.633938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:04.634359 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:04.634592 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.634605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.635687 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.635690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.636453 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.636462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.637126 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.637130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.637280 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.637888 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:04.637904 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:04.639710 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:04.641780 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.641789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.641800 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.642039 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.642338 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.642341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.642343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.642448 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.643027 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:04.643034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:04.643177 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 15:21:04.643330 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 15:21:04.643714 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":160,"thread_start_us":148,"threads_started":1}
I20260927 15:21:04.644713 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.632706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.9.242.190:35631)
0927 15:21:04.633093 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:04.633104 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:04.633148 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:21:04.633542 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:04.633546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:04.633758 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:04.633925 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:04.634722 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.635578 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.636574 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.636984 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.637126 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.639901 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:04.639910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:04.639926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:04.639954 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:04.641608 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.642132 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.642138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.642140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.642258 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.642525 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.642527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.642529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.642926 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:04.643041 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:04.643412 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 15:21:04.643657 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":293,"thread_start_us":206,"threads_started":1}
I20260927 15:21:04.647147 32539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:04.647891 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.647642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50282)
0927 15:21:04.647755 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.647826 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:21:04.648113 32489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
I20260927 15:21:04.648907 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.648697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50296)
0927 15:21:04.648805 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.648854 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:21:04.649024 32539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:37693: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab.
Entry for principal kudu/127.9.242.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab.
I20260927 15:21:04.652465 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:37693
--webserver_interface=127.9.242.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab
--principal=kudu/127.9.242.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:35631,127.9.242.189:37693,127.9.242.188:44845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:21:04.698251 32539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } attempt: 1
I20260927 15:21:04.699134 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.698806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50312)
0927 15:21:04.698965 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.699041 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 15:21:04.699813 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.699595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50324)
0927 15:21:04.699686 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.699738 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 15:21:04.699944 32539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:37693: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
W20260927 15:21:04.760962 32546 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:21:04.761246 32546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:04.761276 32546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:04.761313 32546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:04.765229 32546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:04.765290 32546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:04.765308 32546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:04.765324 32546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:04.765337 32546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:04.768095 32539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } attempt: 2
I20260927 15:21:04.768898 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.768594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50326)
0927 15:21:04.768741 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.768813 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 15:21:04.769600 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.769404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50338)
0927 15:21:04.769492 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.769540 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 15:21:04.769734 32539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:37693: Network error: Client connection negotiation failed: client connection to 127.9.242.189:37693: connect: Connection refused (error 111)
I20260927 15:21:04.771060 32546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:35631,127.9.242.189:37693,127.9.242.188:44845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab
--principal=kudu/127.9.242.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:37693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.242.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:04.772018 32546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:04.772964 32546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:04.781046 32551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:04.781088 32554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:04.781088 32552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:04.781298 32546 server_base.cc:1034] running on GCE node
I20260927 15:21:04.781965 32546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:04.783118 32546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:04.784303 32546 hybrid_clock.cc:648] HybridClock initialized: now 1790522464784286 us; error 34 us; skew 500 ppm
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:04.786895 32546 init.cc:377] Logged in from keytab as kudu/127.9.242.189@KRBTEST.COM (short username kudu)
I20260927 15:21:04.787755 32546 webserver.cc:460] Webserver started at http://127.9.242.189:35985/ using document root <none> and password file <none>
I20260927 15:21:04.788269 32546 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:04.788323 32546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:04.788476 32546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:04.790448 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data/instance:
uuid: "21d95ac61cb4434ab8427bf7a0c99c9f"
format_stamp: "Formatted at 2026-09-27 15:21:04 on dist-test-slave-m4lt"
server_key: "068e43617fef7850ccf8d5fb21977e1e"
server_key_iv: "ac5e0cb28fd904eeb167e0136b2a4cc3"
server_key_version: "encryptionkey@0"
I20260927 15:21:04.790968 32546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/wal/instance:
uuid: "21d95ac61cb4434ab8427bf7a0c99c9f"
format_stamp: "Formatted at 2026-09-27 15:21:04 on dist-test-slave-m4lt"
server_key: "068e43617fef7850ccf8d5fb21977e1e"
server_key_iv: "ac5e0cb28fd904eeb167e0136b2a4cc3"
server_key_version: "encryptionkey@0"
I20260927 15:21:04.794766 32546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:21:04.796994 32561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:04.798339 32546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:21:04.798441 32546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/wal
uuid: "21d95ac61cb4434ab8427bf7a0c99c9f"
format_stamp: "Formatted at 2026-09-27 15:21:04 on dist-test-slave-m4lt"
server_key: "068e43617fef7850ccf8d5fb21977e1e"
server_key_iv: "ac5e0cb28fd904eeb167e0136b2a4cc3"
server_key_version: "encryptionkey@0"
I20260927 15:21:04.798542 32546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:04.825037 32546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:04.828132 32546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:04.828312 32546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:04.835045 32546 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.189:37693
I20260927 15:21:04.835295 32613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.189:37693 every 8 connection(s)
I20260927 15:21:04.835990 32546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
I20260927 15:21:04.838310 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 32546
I20260927 15:21:04.838428 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-1/wal/instance
I20260927 15:21:04.838701 10186 external_mini_cluster.cc:1428] Setting key 2ca4694b55c5527ae6d2ffd10bbd5434
I20260927 15:21:04.838949 32614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:04.843839 32614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.242.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.188@KRBTEST.COM" created.
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:04.858281 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.846186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.9.242.190:35631)
0927 15:21:04.846340 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:04.846345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:04.847214 (+ 869us) server_negotiation.cc:407] Connection header received
0927 15:21:04.848216 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:04.848219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:04.848264 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:04.848351 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:04.849752 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.850322 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.851797 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.852045 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.852142 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.854574 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:04.854577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:04.854579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:04.854606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:04.856062 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.856667 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.856670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.856672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.856749 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.857069 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.857079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.857081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.857283 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:04.857369 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:04.857965 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 15:21:04.858100 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 15:21:04.859153 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.846520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35631 (local address 127.0.0.1:40284)
0927 15:21:04.847033 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.847085 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:04.847955 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:04.848575 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:04.848582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:04.848932 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:04.849571 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.849589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.850456 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.850460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.851659 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.851681 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.852205 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.852208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.852362 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.852923 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:04.852940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:04.854398 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:04.856260 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.856266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.856283 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.856539 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.856836 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.856839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.856841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.856977 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.857386 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:04.857393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:04.857715 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 15:21:04.857895 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 15:21:04.858285 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":149,"threads_started":1}
I20260927 15:21:04.860580 32614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:04.860862 32539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab.
Entry for principal kudu/127.9.242.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab.
I20260927 15:21:04.864868 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:44845
--webserver_interface=127.9.242.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab
--principal=kudu/127.9.242.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:35175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:35631,127.9.242.189:37693,127.9.242.188:44845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
I20260927 15:21:04.871775 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.861118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50350)
0927 15:21:04.861286 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.861299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:04.861373 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:04.862494 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:04.862497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:04.862680 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:04.862786 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.862792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.863806 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.863809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.864680 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.864691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.865546 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.865549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.865716 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.866150 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:04.866164 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:04.867536 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:04.870122 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.870125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.870126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.870343 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.870617 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.870624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.870627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.870701 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.871262 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:04.871265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:04.871347 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 15:21:04.871445 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 15:21:04.871579 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":28}
I20260927 15:21:04.871784 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.861213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50366)
0927 15:21:04.861350 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.861363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:04.861427 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:04.862090 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:04.862093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:04.862252 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:04.862369 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.862375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.863798 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.863801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.865059 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:04.865069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.865581 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.865585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:04.865721 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.866442 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:04.866453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:04.867928 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:04.869976 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.869979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.869981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.870220 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.870544 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:04.870546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:04.870548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:04.870639 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:04.871271 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:04.871274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:04.871374 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 15:21:04.871469 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 15:21:04.871605 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 15:21:04.872522 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.861566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.9.242.189:37693)
0927 15:21:04.861771 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:04.861778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:04.861788 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:21:04.861813 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:04.861815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:04.861950 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:04.862080 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:04.862476 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.863695 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.865176 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.865466 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.865570 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.868070 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:04.868074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:04.868076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:04.868100 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:04.869852 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.870345 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.870351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.870353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.870439 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.870747 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.870749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.870751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.871099 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:04.871224 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:04.872122 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 15:21:04.872354 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":152,"spinlock_wait_cycles":17792,"thread_start_us":63,"threads_started":1}
I20260927 15:21:04.872467 32619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.861293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.9.242.189:37693)
0927 15:21:04.861611 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:04.861620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:04.861637 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:21:04.861700 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:04.861740 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:04.861950 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:04.862080 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:04.862882 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.863705 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.864793 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:04.865329 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:04.865422 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:04.867659 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:04.867671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:04.867688 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:04.867711 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:04.870012 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.870442 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.870445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.870447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.870522 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:04.870798 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:04.870801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:04.870802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:04.871095 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:04.871225 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:04.872118 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 15:21:04.872283 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":241,"spinlock_wait_cycles":10752,"thread_start_us":143,"threads_started":1}
I20260927 15:21:04.874837 32539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:04.874888 32614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:04.875574 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.875295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55484)
0927 15:21:04.875433 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.875494 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 15:21:04.875917 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.875295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55492)
0927 15:21:04.875768 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.875850 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
W20260927 15:21:04.876106 32563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
I20260927 15:21:04.876325 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.876045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55500)
0927 15:21:04.876208 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.876259 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260927 15:21:04.876614 32539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:44845: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
I20260927 15:21:04.877099 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.876881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55512)
0927 15:21:04.876991 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.877041 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:21:04.877249 32614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:44845: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
I20260927 15:21:04.907784 32539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } attempt: 1
I20260927 15:21:04.908665 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.908293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55526)
0927 15:21:04.908475 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.908560 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 15:21:04.909412 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.909162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55536)
0927 15:21:04.909278 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.909337 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 15:21:04.909554 32539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:44845: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
I20260927 15:21:04.926503 32614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } attempt: 1
I20260927 15:21:04.927382 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.927056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55544)
0927 15:21:04.927214 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.927289 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 15:21:04.928145 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.927939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55560)
0927 15:21:04.928033 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.928083 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 15:21:04.928280 32614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:44845: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
W20260927 15:21:04.978518 32621 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:21:04.978775 32621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:04.978796 32621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:04.978840 32621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:04.982854 32621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:04.982910 32621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:04.982928 32621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:04.982944 32621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:04.982967 32621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:04.984722 32539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } attempt: 2
I20260927 15:21:04.985523 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.985202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55568)
0927 15:21:04.985367 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.985434 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 15:21:04.986171 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.985969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55574)
0927 15:21:04.986061 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.986110 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 15:21:04.986284 32539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:44845: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
I20260927 15:21:04.988548 32621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:35175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:35631,127.9.242.189:37693,127.9.242.188:44845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab
--principal=kudu/127.9.242.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:44845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.242.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:04.989535 32621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:04.990540 32621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:21:04.996445 32614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } attempt: 2
I20260927 15:21:04.997269 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.996934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55586)
0927 15:21:04.997094 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.997175 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 15:21:04.998009 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:04.997765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55594)
0927 15:21:04.997864 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:04.997936 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 15:21:04.998322 32614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:44845: Network error: Client connection negotiation failed: client connection to 127.9.242.188:44845: connect: Connection refused (error 111)
W20260927 15:21:04.999243 32628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:04.999258 32627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:04.999281 32630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:04.999962 32621 server_base.cc:1034] running on GCE node
I20260927 15:21:05.000535 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:05.001843 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:05.003005 32621 hybrid_clock.cc:648] HybridClock initialized: now 1790522465002983 us; error 36 us; skew 500 ppm
Sep 27 15:21:04 dist-test-slave-m4lt krb5kdc[32458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:05.005847 32621 init.cc:377] Logged in from keytab as kudu/127.9.242.188@KRBTEST.COM (short username kudu)
I20260927 15:21:05.006737 32621 webserver.cc:460] Webserver started at http://127.9.242.188:39769/ using document root <none> and password file <none>
I20260927 15:21:05.007274 32621 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:05.007333 32621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:05.007521 32621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:05.009564 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data/instance:
uuid: "3c29401cd421418c97801afc4715d028"
format_stamp: "Formatted at 2026-09-27 15:21:05 on dist-test-slave-m4lt"
server_key: "2eabb092f4e9b25d5f8c1c512fc139fd"
server_key_iv: "ddc7ce804949ec52ce40971d4cceaa27"
server_key_version: "encryptionkey@0"
I20260927 15:21:05.010174 32621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/wal/instance:
uuid: "3c29401cd421418c97801afc4715d028"
format_stamp: "Formatted at 2026-09-27 15:21:05 on dist-test-slave-m4lt"
server_key: "2eabb092f4e9b25d5f8c1c512fc139fd"
server_key_iv: "ddc7ce804949ec52ce40971d4cceaa27"
server_key_version: "encryptionkey@0"
I20260927 15:21:05.014060 32621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 15:21:05.016506 32637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:05.017939 32621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 15:21:05.018059 32621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/wal
uuid: "3c29401cd421418c97801afc4715d028"
format_stamp: "Formatted at 2026-09-27 15:21:05 on dist-test-slave-m4lt"
server_key: "2eabb092f4e9b25d5f8c1c512fc139fd"
server_key_iv: "ddc7ce804949ec52ce40971d4cceaa27"
server_key_version: "encryptionkey@0"
I20260927 15:21:05.018162 32621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:05.031744 32621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:05.035162 32621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:05.035346 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:05.042138 32621 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.188:44845
I20260927 15:21:05.042443 32689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.188:44845 every 8 connection(s)
I20260927 15:21:05.043114 32621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
I20260927 15:21:05.045966 32690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:05.050066 32690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:05.051806 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 32621
I20260927 15:21:05.051934 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790522194092531-10186-0/minicluster-data/master-2/wal/instance
I20260927 15:21:05.052244 10186 external_mini_cluster.cc:1428] Setting key 04819ab8dec3987775a6367b05eb13d7
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:21:05.064255 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.051996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.9.242.190:35631)
0927 15:21:05.052156 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.052161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.053119 (+ 958us) server_negotiation.cc:407] Connection header received
0927 15:21:05.054204 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.054207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.054294 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.054394 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.055451 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.056208 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.057603 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.057976 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.058077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.060815 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.060819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.060822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.060849 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.062263 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.062793 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.062796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.062798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.062874 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.063190 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.063193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.063195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.063385 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.063466 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.063953 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.064089 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 15:21:05.065016 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.052414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:35631 (local address 127.0.0.1:40296)
0927 15:21:05.052954 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:05.053017 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:05.054012 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:05.054472 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:05.054478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:05.054912 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:05.055322 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.055335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.056328 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.056331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.057473 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.057484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.058117 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.058120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.058320 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.059017 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:05.059037 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:05.060612 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:05.062359 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.062367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.062378 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.062704 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.062963 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.062966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.062968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.063104 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.063480 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:05.063487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:05.063728 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 15:21:05.063876 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 15:21:05.064178 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":3,"spinlock_wait_cycles":9984,"thread_start_us":132,"threads_started":1}
I20260927 15:21:05.065016 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.052903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.9.242.188:44845)
0927 15:21:05.053270 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.053281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.053317 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:21:05.054024 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.054046 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.054260 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.054436 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.055252 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.056476 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.057603 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.058228 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.058315 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.061154 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.061159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.061169 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.061188 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.062694 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.063068 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.063084 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.063085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.063156 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.063399 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.063402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.063403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.063731 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.063909 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.064107 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.064252 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":5632,"thread_start_us":122,"threads_started":1}
I20260927 15:21:05.066159 32690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:05.074806 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.066623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:37693 (local address 127.0.0.1:50370)
0927 15:21:05.066772 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:05.066786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:05.066858 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:05.067088 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:05.067091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:05.067249 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:05.067359 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.067366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.068360 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.068363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.069172 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.069181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.069849 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.069852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.069935 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.070346 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:05.070361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:05.071606 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:05.073258 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.073262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.073264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.073492 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.073834 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.073837 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.073839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.073928 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.074364 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:05.074368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:05.074467 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 15:21:05.074544 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 15:21:05.074649 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 15:21:05.074885 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.066722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.9.242.189:37693)
0927 15:21:05.066899 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.066903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.066917 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:21:05.066959 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.066962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.067003 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.067064 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.067474 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.068258 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.069339 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.069739 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.069841 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.071761 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.071766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.071768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.071801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.073157 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.073583 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.073586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.073588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.073745 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.074044 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.074048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.074049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.074264 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.074356 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.074631 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.074758 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 15:21:05.075781 32690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:05.076045 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.067327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.9.242.190:35631)
0927 15:21:05.067438 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.067442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.067473 (+ 31us) server_negotiation.cc:407] Connection header received
0927 15:21:05.067575 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.067577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.067610 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.067662 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.068311 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.069117 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.070110 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.070390 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.070487 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.072803 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.072806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.072808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.072830 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.074311 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.074768 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.074771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.074773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.074851 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.075120 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.075124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.075125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.075294 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.075519 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.075794 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.075898 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
I20260927 15:21:05.083413 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.076251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55604)
0927 15:21:05.076383 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:05.076398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:05.076471 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:05.076746 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:05.076748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:05.076870 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:05.076942 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.076947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.077842 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.077845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.078346 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.078353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.078771 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.078774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.078843 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.079277 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:05.079290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:05.080548 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:05.082150 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.082153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.082156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.082322 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.082538 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.082541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.082543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.082619 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.082931 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:05.082934 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:05.083012 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 15:21:05.083104 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:21:05.083235 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 15:21:05.083413 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.076366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.9.242.188:44845)
0927 15:21:05.076465 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.076468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.076479 (+ 11us) server_negotiation.cc:407] Connection header received
0927 15:21:05.076544 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.076547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.076586 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.076651 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.077038 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.077760 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.078438 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.078685 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.078765 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.080662 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.080665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.080668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.080689 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.082052 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.082397 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.082400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.082403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.082469 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.082687 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.082690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.082691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.082848 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.082932 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.083164 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.083280 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":35}
I20260927 15:21:05.085419 32539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } attempt: 3
I20260927 15:21:05.086273 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.077133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.9.242.189:37693)
0927 15:21:05.077237 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.077241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.077254 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:21:05.077306 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.077308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.077346 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.077416 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.077954 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.078425 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.079133 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.079443 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.079540 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.081613 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.081617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.081620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.081644 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.082983 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.083348 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.083352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.083354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.083423 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.083961 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.083964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.083966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.084132 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.085799 (+ 1667us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.085873 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.085991 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:21:05.087091 32690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: Bootstrap starting.
I20260927 15:21:05.089381 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:21:05.089463 32614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } attempt: 3
I20260927 15:21:05.089808 32690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: Neither blocks nor log segments found. Creating new log.
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:21:05.090682 32690 log.cc:824] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:05.091001 32513 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c8f67288af1943308a6f7daf1fcd23fb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40306
I20260927 15:21:05.094308 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.085857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55618)
0927 15:21:05.085982 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:05.086003 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:05.086081 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:05.086666 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:05.086669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:05.086830 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:05.086929 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.086936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.087827 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.087830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.088582 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.088594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.089095 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.089107 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.089195 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.089723 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:05.089738 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:05.091064 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:05.092826 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.092830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.092832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.093076 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.093317 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.093320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.093322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.093427 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.093881 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:05.093884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:05.094000 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 15:21:05.094084 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 15:21:05.094195 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 15:21:05.094575 32690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: No bootstrap required, opened a new log
I20260927 15:21:05.094846 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.086074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.9.242.188:44845)
0927 15:21:05.086450 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.086454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.086466 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:21:05.086494 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.086497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.086534 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.086643 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.087041 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.087690 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.088684 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.088987 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.089072 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.091231 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.091235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.091238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.091261 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.092731 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.093147 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.093151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.093153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.093227 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.093524 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.093527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.093529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.093782 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.093873 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.094187 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.094613 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
Sep 27 15:21:05 dist-test-slave-m4lt krb5kdc[32458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522464, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:21:05.097769 32539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb: Bootstrap starting.
I20260927 15:21:05.098634 32690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
I20260927 15:21:05.098874 32690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c29401cd421418c97801afc4715d028, State: Initialized, Role: FOLLOWER
I20260927 15:21:05.099115 32617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.089811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:44845 (local address 127.0.0.1:55620)
0927 15:21:05.090457 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:05.090473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:05.090546 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:05.091182 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:05.091185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:05.091356 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:05.091434 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.091439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.092263 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.092266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.092888 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:05.092898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.093514 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.093517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:05.093599 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.094467 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:05.094482 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:05.095782 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:05.097615 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.097618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.097634 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.097894 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.098154 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:05.098157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:05.098159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:05.098265 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:05.098632 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:05.098634 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:05.098728 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 15:21:05.098840 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 15:21:05.098969 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575}
I20260927 15:21:05.099207 32690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
I20260927 15:21:05.099556 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.090279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.9.242.188:44845)
0927 15:21:05.090909 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.090917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.090930 (+ 13us) server_negotiation.cc:407] Connection header received
0927 15:21:05.090961 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.090963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.091001 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.091517 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.091596 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.092125 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.093009 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.093419 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.093500 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.095914 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.095918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.095920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.095943 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.097511 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.097985 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.097988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.097990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.098074 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.098351 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.098355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.098357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.098543 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.098637 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.099024 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.099132 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":90,"threads_started":1}
I20260927 15:21:05.100320 32539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:05.100731 32690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:05.100948 32539 log.cc:824] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:05.102288 32614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: Bootstrap starting.
I20260927 15:21:05.102910 32539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb: No bootstrap required, opened a new log
I20260927 15:21:05.104506 32614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:05.105330 32614 log.cc:824] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:05.103099 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
W20260927 15:21:05.105505 32709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 15:21:05.105715 32539 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
W20260927 15:21:05.105859 32709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:21:05.105856 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:05.105912 32539 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f67288af1943308a6f7daf1fcd23fb, State: Initialized, Role: FOLLOWER
I20260927 15:21:05.106170 32539 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
I20260927 15:21:05.107343 32539 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:05.108161 32614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: No bootstrap required, opened a new log
I20260927 15:21:05.107199 32621 master_runner.cc:186] Error getting master registration for 127.9.242.190:35631: OK, instance_id { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" instance_seqno: 1790522464619758 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:21:05.110368 32621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:21:05.110556 32513 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:21:05.110657 32513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:05.111816 32513 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
I20260927 15:21:05.111909 32614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
I20260927 15:21:05.112102 32614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d95ac61cb4434ab8427bf7a0c99c9f, State: Initialized, Role: FOLLOWER
I20260927 15:21:05.112414 32614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
I20260927 15:21:05.112679 32513 leader_election.cc:288] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [CANDIDATE]: Term 1 election: Requested vote from peers 21d95ac61cb4434ab8427bf7a0c99c9f (127.9.242.189:37693), 3c29401cd421418c97801afc4715d028 (127.9.242.188:44845)
W20260927 15:21:05.112926 32723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:21:05.112984 32723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:21:05.113440 32664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8f67288af1943308a6f7daf1fcd23fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c29401cd421418c97801afc4715d028"
I20260927 15:21:05.113564 32614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:05.113639 32664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:05.111572 32471 master_runner.cc:186] Error getting master registration for 127.9.242.189:37693: OK, instance_id { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" instance_seqno: 1790522464833131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:21:05.114539 32471 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:21:05.114881 32664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f67288af1943308a6f7daf1fcd23fb in term 1.
I20260927 15:21:05.115010 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8f67288af1943308a6f7daf1fcd23fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f"
I20260927 15:21:05.115186 32587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:05.116068 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.116181 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:05.116389 32587 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f67288af1943308a6f7daf1fcd23fb in term 1.
W20260927 15:21:05.116600 32492 tablet.cc:2369] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:21:05.116770 32489 leader_election.cc:302] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d95ac61cb4434ab8427bf7a0c99c9f, c8f67288af1943308a6f7daf1fcd23fb; no voters:
I20260927 15:21:05.117952 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.118081 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:05.118242 32712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 15:21:05.118449 32712 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [term 1 LEADER]: Becoming Leader. State: Replica: c8f67288af1943308a6f7daf1fcd23fb, State: Running, Role: LEADER
I20260927 15:21:05.118753 32712 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } }
W20260927 15:21:05.120632 32740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:21:05.120708 32740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:21:05.127702 32741 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f67288af1943308a6f7daf1fcd23fb. Latest consensus state: current_term: 1 leader_uuid: "c8f67288af1943308a6f7daf1fcd23fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.127869 32741 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:05.128551 32747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:21:05.129769 32747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:21:05.135661 32587 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [term 1 FOLLOWER]: Refusing update from remote peer c8f67288af1943308a6f7daf1fcd23fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:05.135704 32664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [term 1 FOLLOWER]: Refusing update from remote peer c8f67288af1943308a6f7daf1fcd23fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:05.136312 32741 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:05.136704 32741 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:05.140091 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f67288af1943308a6f7daf1fcd23fb. Latest consensus state: current_term: 1 leader_uuid: "c8f67288af1943308a6f7daf1fcd23fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.140215 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:05.140341 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f67288af1943308a6f7daf1fcd23fb. Latest consensus state: current_term: 1 leader_uuid: "c8f67288af1943308a6f7daf1fcd23fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.140432 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:05.149524 32620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.126332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.9.242.189:37693)
0927 15:21:05.126452 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.126457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.140739 (+ 14282us) server_negotiation.cc:407] Connection header received
0927 15:21:05.140845 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.140848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.140904 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.140984 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.141949 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.142739 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.143938 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.144274 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.144342 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.146139 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.146143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.146145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.146170 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.147686 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.148148 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.148152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.148153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.148233 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.148515 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.148518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.148520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.148809 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.148912 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.149244 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.149371 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 15:21:05.151607 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8f67288af1943308a6f7daf1fcd23fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.151805 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:05.151999 32748 mvcc.cc:205] Tried to move back new op lower bound from 7333980017189806080 to 7333980017157320704. Current Snapshot: MvccSnapshot[applied={T|T < 7333980017189806080}]
I20260927 15:21:05.152477 32725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8f67288af1943308a6f7daf1fcd23fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.152588 32725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:05.155223 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.126714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.9.242.188:44845)
0927 15:21:05.126823 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.126827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.134698 (+ 7871us) server_negotiation.cc:407] Connection header received
0927 15:21:05.134885 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.134888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.134937 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.134984 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.145752 (+ 10768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.146554 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.149224 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.149593 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.149691 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.151383 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.151387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.151390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.151417 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.153067 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.153522 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.153525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.153527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.153615 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.153960 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.153963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.153965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.154175 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.154304 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.154934 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.155050 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:21:05.155875 32750 mvcc.cc:205] Tried to move back new op lower bound from 7333980017189806080 to 7333980017157320704. Current Snapshot: MvccSnapshot[applied={T|T < 7333980017189806080}]
I20260927 15:21:05.165253 32747 catalog_manager.cc:1347] Generated new cluster ID: 980d963b9315440ca12b13232e58d4a3
I20260927 15:21:05.165333 32747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:21:05.170246 32712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8f67288af1943308a6f7daf1fcd23fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.170394 32712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:05.171643 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8f67288af1943308a6f7daf1fcd23fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f67288af1943308a6f7daf1fcd23fb" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 35631 } } peers { permanent_uuid: "21d95ac61cb4434ab8427bf7a0c99c9f" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 37693 } } peers { permanent_uuid: "3c29401cd421418c97801afc4715d028" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 44845 } } }
I20260927 15:21:05.171775 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:05.174105 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:05.126093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.9.242.190:35631)
0927 15:21:05.126226 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.126231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.149194 (+ 22963us) server_negotiation.cc:407] Connection header received
0927 15:21:05.149367 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.149370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.149419 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.149466 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.153213 (+ 3747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.153872 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.154888 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:05.155213 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:05.155286 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:05.158703 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:05.158707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:05.158710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:05.158737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:05.160234 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.172282 (+ 12048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.172286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.172288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.172402 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:05.172737 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:05.172740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:05.172742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:05.172974 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:05.173111 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:05.173816 (+ 705us) server_negotiation.cc:299] Negotiation successful
0927 15:21:05.173946 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:21:05.177912 32749 mvcc.cc:205] Tried to move back new op lower bound from 7333980017189806080 to 7333980017157320704. Current Snapshot: MvccSnapshot[applied={T|T < 7333980017189806080}]
I20260927 15:21:05.214648 32747 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:21:05.216109 32747 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:21:05.252683 32747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f67288af1943308a6f7daf1fcd23fb: Generated new TSK 0
I20260927 15:21:05.255911 32747 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:21:05.293639 32543 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.290567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.9.242.190:35631)
0927 15:21:05.290744 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.290749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.292537 (+ 1788us) server_negotiation.cc:407] Connection header received
0927 15:21:05.292745 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.292750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.292802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.292855 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.293497 (+ 642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40332: BlockingRecv error: recv got EOF from 127.0.0.1:40332 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 15:21:05.295085 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.291494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.9.242.189:37693)
0927 15:21:05.291625 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.291629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.294158 (+ 2529us) server_negotiation.cc:407] Connection header received
0927 15:21:05.294350 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.294353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.294404 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.294456 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.294964 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50404: BlockingRecv error: recv got EOF from 127.0.0.1:50404 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 15:21:05.296368 32695 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.292007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.9.242.188:44845)
0927 15:21:05.292107 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.292111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.295629 (+ 3518us) server_negotiation.cc:407] Connection header received
0927 15:21:05.295748 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.295752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.295801 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.295873 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.296254 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55638: BlockingRecv error: recv got EOF from 127.0.0.1:55638 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 15:21:05.300530 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.298952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.9.242.189:37693)
0927 15:21:05.299065 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.299069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.299552 (+ 483us) server_negotiation.cc:407] Connection header received
0927 15:21:05.299766 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.299770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.299848 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.299905 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.300426 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50418: BlockingRecv error: recv got EOF from 127.0.0.1:50418 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 15:21:05.305565 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.304098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.9.242.189:37693)
0927 15:21:05.304207 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.304211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.304766 (+ 555us) server_negotiation.cc:407] Connection header received
0927 15:21:05.304932 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.304935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.304981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.305026 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.305469 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50426: BlockingRecv error: recv got EOF from 127.0.0.1:50426 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 15:21:05.311095 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.309641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.9.242.189:37693)
0927 15:21:05.309760 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.309764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.310253 (+ 489us) server_negotiation.cc:407] Connection header received
0927 15:21:05.310422 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.310425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.310471 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.310517 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.310989 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50434: BlockingRecv error: recv got EOF from 127.0.0.1:50434 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 15:21:05.318902 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.317228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.9.242.189:37693)
0927 15:21:05.317361 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.317365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.317985 (+ 620us) server_negotiation.cc:407] Connection header received
0927 15:21:05.318165 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.318180 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.318231 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.318281 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.318784 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50436: BlockingRecv error: recv got EOF from 127.0.0.1:50436 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 15:21:05.331193 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.329274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.9.242.189:37693)
0927 15:21:05.329399 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.329404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.330096 (+ 692us) server_negotiation.cc:407] Connection header received
0927 15:21:05.330328 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.330332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.330390 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.330444 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.331075 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50444: BlockingRecv error: recv got EOF from 127.0.0.1:50444 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 15:21:05.350118 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.348964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.9.242.189:37693)
0927 15:21:05.349077 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.349080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.349228 (+ 148us) server_negotiation.cc:407] Connection header received
0927 15:21:05.349374 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.349376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.349425 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.349487 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.350004 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50452: BlockingRecv error: recv got EOF from 127.0.0.1:50452 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 15:21:05.385258 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.384140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.9.242.189:37693)
0927 15:21:05.384311 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.384315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.384420 (+ 105us) server_negotiation.cc:407] Connection header received
0927 15:21:05.384580 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.384583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.384633 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.384702 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.385155 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50468: BlockingRecv error: recv got EOF from 127.0.0.1:50468 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 15:21:05.452450 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.451342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.9.242.189:37693)
0927 15:21:05.451519 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.451523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.451643 (+ 120us) server_negotiation.cc:407] Connection header received
0927 15:21:05.451733 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.451735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.451786 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.451863 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.452331 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50484: BlockingRecv error: recv got EOF from 127.0.0.1:50484 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 15:21:05.583580 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.582464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.9.242.189:37693)
0927 15:21:05.582616 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.582621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.582713 (+ 92us) server_negotiation.cc:407] Connection header received
0927 15:21:05.582894 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.582898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.582945 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.583010 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.583461 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50486: BlockingRecv error: recv got EOF from 127.0.0.1:50486 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 15:21:05.842779 32620 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:05.841683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.9.242.189:37693)
0927 15:21:05.841820 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:05.841825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:05.841925 (+ 100us) server_negotiation.cc:407] Connection header received
0927 15:21:05.842065 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:05.842068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:05.842116 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:05.842185 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:05.842659 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36404: BlockingRecv error: recv got EOF from 127.0.0.1:36404 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 15:21:06.107283 32709 catalog_manager.cc:1259] Loaded cluster ID: 980d963b9315440ca12b13232e58d4a3
I20260927 15:21:06.107364 32709 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: loading cluster ID for follower catalog manager: success
I20260927 15:21:06.109086 32709 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: acquiring CA information for follower catalog manager: success
I20260927 15:21:06.110250 32709 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c29401cd421418c97801afc4715d028: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:21:06.121960 32740 catalog_manager.cc:1259] Loaded cluster ID: 980d963b9315440ca12b13232e58d4a3
I20260927 15:21:06.122030 32740 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: loading cluster ID for follower catalog manager: success
I20260927 15:21:06.123713 32740 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: acquiring CA information for follower catalog manager: success
I20260927 15:21:06.124876 32740 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21d95ac61cb4434ab8427bf7a0c99c9f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:21:06.360751 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:06.356707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.9.242.189:37693)
0927 15:21:06.356986 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:06.356989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:06.357006 (+ 17us) server_negotiation.cc:407] Connection header received
0927 15:21:06.357125 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:06.357128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:06.357305 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:06.357412 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:06.357681 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.358260 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.359182 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.359429 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.359517 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:06.359768 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:06.360289 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:06.360390 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:06.360533 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 15:21:06.360587 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20260927 15:21:06.367385 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:06.363915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.9.242.188:44845)
0927 15:21:06.364136 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:06.364141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:06.364155 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:21:06.364276 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:06.364279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:06.364392 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:06.364498 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:06.364695 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.365201 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.366044 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.366270 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.366358 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:06.366611 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:06.367019 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:06.367124 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:06.367185 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 15:21:06.367245 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260927 15:21:06.373858 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:06.370421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.9.242.188:44845)
0927 15:21:06.370518 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:06.370521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:06.370640 (+ 119us) server_negotiation.cc:407] Connection header received
0927 15:21:06.370803 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:06.370806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:06.370853 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:06.370893 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:06.371276 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.371976 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.372804 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.373064 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.373150 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:06.373388 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:06.373465 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:06.373517 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:06.373646 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 15:21:06.373718 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 15:21:06.374303 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:06.370096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.9.242.189:37693)
0927 15:21:06.370210 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:06.370214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:06.370704 (+ 490us) server_negotiation.cc:407] Connection header received
0927 15:21:06.370811 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:06.370814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:06.370865 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:06.370927 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:06.371307 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.371794 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.372993 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.373292 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.373377 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:06.373832 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:06.373909 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:06.373953 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:06.374124 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 15:21:06.374180 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 15:21:06.374655 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:06.370083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.9.242.190:35631)
0927 15:21:06.370500 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:06.370504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:06.370518 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:21:06.370609 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:06.370612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:06.370777 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:06.370887 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:06.371082 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.371864 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.373027 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:06.373328 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:06.373411 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:06.373793 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:06.374105 (+ 312us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:06.374196 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:06.374460 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 15:21:06.374526 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":65,"threads_started":1}
I20260927 15:21:06.381330 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 32471
I20260927 15:21:06.386912 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 32546
I20260927 15:21:06.391878 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 32621
2026-09-27T15:21:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3025 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 15:21:06 dist-test-slave-m4lt krb5kdc[370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 15:21:06 dist-test-slave-m4lt krb5kdc[370](info): set up 2 sockets
Sep 27 15:21:06 dist-test-slave-m4lt krb5kdc[370](info): commencing operation
krb5kdc: starting...
W20260927 15:21:07.349795 10186 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/test-user.keytab.
Sep 27 15:21:07 dist-test-slave-m4lt krb5kdc[370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T15:21:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T15:21:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.242.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
Entry for principal kudu/127.9.242.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab.
I20260927 15:21:07.466585 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:33457
--webserver_interface=127.9.242.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:33457,127.9.242.189:43205,127.9.242.188:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 15:21:07.577280 383 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:21:07.577576 383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:07.577600 383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:07.577677 383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:07.581710 383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:07.581771 383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:07.581790 383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:07.581805 383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:07.581826 383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:07.587299 383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:33457,127.9.242.189:43205,127.9.242.188:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.190.keytab
--principal=kudu/127.9.242.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.190:33457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.242.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:07.588442 383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:07.589452 383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:07.597818 391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:07.597818 388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:07.597898 389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:07.597893 383 server_base.cc:1034] running on GCE node
I20260927 15:21:07.598592 383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:07.599690 383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:07.600853 383 hybrid_clock.cc:648] HybridClock initialized: now 1790522467600839 us; error 28 us; skew 500 ppm
Sep 27 15:21:07 dist-test-slave-m4lt krb5kdc[370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:07.603510 383 init.cc:377] Logged in from keytab as kudu/127.9.242.190@KRBTEST.COM (short username kudu)
I20260927 15:21:07.604354 383 webserver.cc:460] Webserver started at http://127.9.242.190:33913/ using document root <none> and password file <none>
I20260927 15:21:07.604864 383 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:07.604920 383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:07.605085 383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:07.607064 383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data/instance:
uuid: "2b7901c47f364235bf3af953e448eab9"
format_stamp: "Formatted at 2026-09-27 15:21:07 on dist-test-slave-m4lt"
server_key: "7dc1d7b629929398d66d35df799ff5ac"
server_key_iv: "f863fdb32cf8a78e1f3d48e329a73385"
server_key_version: "encryptionkey@0"
I20260927 15:21:07.607564 383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/wal/instance:
uuid: "2b7901c47f364235bf3af953e448eab9"
format_stamp: "Formatted at 2026-09-27 15:21:07 on dist-test-slave-m4lt"
server_key: "7dc1d7b629929398d66d35df799ff5ac"
server_key_iv: "f863fdb32cf8a78e1f3d48e329a73385"
server_key_version: "encryptionkey@0"
I20260927 15:21:07.611112 383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 15:21:07.613368 398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:07.614571 383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:21:07.614708 383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/wal
uuid: "2b7901c47f364235bf3af953e448eab9"
format_stamp: "Formatted at 2026-09-27 15:21:07 on dist-test-slave-m4lt"
server_key: "7dc1d7b629929398d66d35df799ff5ac"
server_key_iv: "f863fdb32cf8a78e1f3d48e329a73385"
server_key_version: "encryptionkey@0"
I20260927 15:21:07.614807 383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:07.634008 383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:07.636885 383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:07.637058 383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:07.643711 383 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.190:33457
I20260927 15:21:07.643950 450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.190:33457 every 8 connection(s)
I20260927 15:21:07.644692 383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/data/info.pb
I20260927 15:21:07.647478 451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:07.651207 451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:07.652405 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 383
I20260927 15:21:07.652519 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-0/wal/instance
I20260927 15:21:07.652786 10186 external_mini_cluster.cc:1428] Setting key 57ebfd9c03b8b9b2fc471ff553b5df86
WARNING: no policy specified for kudu/127.9.242.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.189@KRBTEST.COM" created.
Sep 27 15:21:07 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:07.668396 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.654342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.9.242.190:33457)
0927 15:21:07.654607 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:07.654617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:07.654660 (+ 43us) server_negotiation.cc:407] Connection header received
0927 15:21:07.655295 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:07.655299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:07.655481 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:07.655637 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:07.656475 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.657416 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.658596 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.659003 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.659147 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.663403 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:07.663408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:07.663419 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:07.663441 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:07.665222 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.665718 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.665724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.665725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.665832 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.666127 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.666129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.666131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.666535 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:07.666662 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:07.667025 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 15:21:07.667268 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260927 15:21:07.668396 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.653483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:33457 (local address 127.0.0.1:53192)
0927 15:21:07.654040 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.654105 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:07.655012 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:07.655648 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:07.655653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:07.656086 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:07.656344 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.656358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.657614 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.657618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.658479 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.658490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.659151 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.659155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.659301 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.659838 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:07.659854 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:07.663193 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:07.665330 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.665338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.665349 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.665565 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.665918 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.665921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.665924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.666031 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.666642 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:07.666648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:07.666782 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 15:21:07.666926 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 15:21:07.667325 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":134,"thread_start_us":153,"threads_started":1}
I20260927 15:21:07.671591 451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:07.672415 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.672087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36130)
0927 15:21:07.672204 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.672319 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 15:21:07.672659 399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
I20260927 15:21:07.673746 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.673414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36136)
0927 15:21:07.673541 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.673590 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 15:21:07.673928 451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:43205: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab.
Entry for principal kudu/127.9.242.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab.
I20260927 15:21:07.678033 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:43205
--webserver_interface=127.9.242.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab
--principal=kudu/127.9.242.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:33457,127.9.242.189:43205,127.9.242.188:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 15:21:07.723201 451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } attempt: 1
I20260927 15:21:07.723999 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.723706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36138)
0927 15:21:07.723866 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.723925 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 15:21:07.724705 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.724512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36146)
0927 15:21:07.724598 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.724642 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 15:21:07.724854 451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:43205: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
W20260927 15:21:07.787182 458 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:21:07.787479 458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:07.787501 458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:07.787537 458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 15:21:07.791466 458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:07.791517 458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:07.791533 458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:07.791548 458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:07.791561 458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:07.792999 451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } attempt: 2
I20260927 15:21:07.793849 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.793505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36160)
0927 15:21:07.793672 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.793768 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 15:21:07.794556 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.794372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36168)
0927 15:21:07.794460 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.794504 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 15:21:07.794672 451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.189:43205: Network error: Client connection negotiation failed: client connection to 127.9.242.189:43205: connect: Connection refused (error 111)
I20260927 15:21:07.797209 458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:33457,127.9.242.189:43205,127.9.242.188:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.189.keytab
--principal=kudu/127.9.242.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.189:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.242.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:07.798293 458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:07.799273 458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 15:21:07.807297 464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:07.807338 463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:07.807325 466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:07.807541 458 server_base.cc:1034] running on GCE node
I20260927 15:21:07.808187 458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:07.809233 458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:07.810436 458 hybrid_clock.cc:648] HybridClock initialized: now 1790522467810416 us; error 28 us; skew 500 ppm
Sep 27 15:21:07 dist-test-slave-m4lt krb5kdc[370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:07.813274 458 init.cc:377] Logged in from keytab as kudu/127.9.242.189@KRBTEST.COM (short username kudu)
I20260927 15:21:07.814174 458 webserver.cc:460] Webserver started at http://127.9.242.189:44695/ using document root <none> and password file <none>
I20260927 15:21:07.814718 458 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:07.814779 458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:07.814956 458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:07.816923 458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data/instance:
uuid: "410b17e355d24deaa35e59fa2d588d44"
format_stamp: "Formatted at 2026-09-27 15:21:07 on dist-test-slave-m4lt"
server_key: "e1bc8b7fc4bc000eb3f918624a89ad3c"
server_key_iv: "c4bfa85ac5f1cfc13eb2047c31bb2e84"
server_key_version: "encryptionkey@0"
I20260927 15:21:07.817467 458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/wal/instance:
uuid: "410b17e355d24deaa35e59fa2d588d44"
format_stamp: "Formatted at 2026-09-27 15:21:07 on dist-test-slave-m4lt"
server_key: "e1bc8b7fc4bc000eb3f918624a89ad3c"
server_key_iv: "c4bfa85ac5f1cfc13eb2047c31bb2e84"
server_key_version: "encryptionkey@0"
I20260927 15:21:07.821069 458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 15:21:07.823441 473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:07.824601 458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 15:21:07.824734 458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/wal
uuid: "410b17e355d24deaa35e59fa2d588d44"
format_stamp: "Formatted at 2026-09-27 15:21:07 on dist-test-slave-m4lt"
server_key: "e1bc8b7fc4bc000eb3f918624a89ad3c"
server_key_iv: "c4bfa85ac5f1cfc13eb2047c31bb2e84"
server_key_version: "encryptionkey@0"
I20260927 15:21:07.824822 458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:07.850862 458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:07.854072 458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:07.854264 458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:07.861171 458 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.189:43205
I20260927 15:21:07.861419 525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.189:43205 every 8 connection(s)
I20260927 15:21:07.862244 458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/data/info.pb
I20260927 15:21:07.864266 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 458
I20260927 15:21:07.864392 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-1/wal/instance
I20260927 15:21:07.864691 10186 external_mini_cluster.cc:1428] Setting key cb96a155ee962a2499d3324860a38716
I20260927 15:21:07.865402 526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:07.869354 526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.242.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.242.188@KRBTEST.COM" created.
Sep 27 15:21:07 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:07.883975 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.871306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.9.242.190:33457)
0927 15:21:07.871457 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:07.871462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:07.872321 (+ 859us) server_negotiation.cc:407] Connection header received
0927 15:21:07.873289 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:07.873293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:07.873350 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:07.873424 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:07.874777 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.875297 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.876453 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.876698 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.876800 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.879711 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:07.879716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:07.879719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:07.879748 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:07.881550 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.882224 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.882228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.882230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.882329 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.882719 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.882722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.882723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.882956 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:07.883050 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:07.883679 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 15:21:07.883806 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 15:21:07.884954 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.871679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:33457 (local address 127.0.0.1:53194)
0927 15:21:07.872158 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.872204 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:07.873070 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:07.873605 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:07.873612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:07.874083 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:07.874626 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.874641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.875416 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.875419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.876342 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.876353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.876870 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.876874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.877079 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.877735 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:07.877755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:07.879482 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:07.881722 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.881729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.881746 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.882115 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.882431 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.882435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.882437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.882626 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.883070 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:07.883077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:07.883414 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 15:21:07.883598 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 15:21:07.883947 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260927 15:21:07.885802 451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } attempt: 3
I20260927 15:21:07.886489 526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.242.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab.
Entry for principal kudu/127.9.242.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab.
I20260927 15:21:07.892344 10186 external_mini_cluster.cc:1326] Running /tmp/dist-test-task93l8wz/build/debug/bin/kudu
/tmp/dist-test-task93l8wz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:34765
--webserver_interface=127.9.242.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab
--principal=kudu/127.9.242.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.242.148:34659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.242.190:33457,127.9.242.189:43205,127.9.242.188:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 15:21:07 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
Sep 27 15:21:07 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
I20260927 15:21:07.896736 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.886939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36190)
0927 15:21:07.887028 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.887040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:07.887096 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:07.887801 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:07.887804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:07.887966 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:07.888069 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.888075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.889053 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.889056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.890040 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.890051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.890754 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.890757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.890861 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.891394 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:07.891406 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:07.892867 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:07.895138 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.895142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.895145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.895361 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.895653 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.895657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.895659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.895745 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.896213 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:07.896216 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:07.896304 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 15:21:07.896398 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 15:21:07.896554 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":21}
I20260927 15:21:07.897116 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.886181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36184)
0927 15:21:07.886316 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.886330 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:07.886412 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:07.887130 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:07.887133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:07.887308 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:07.887409 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.887415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.888827 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.888831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.890366 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:07.890378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.891017 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.891021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:07.891126 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.891581 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:07.891595 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:07.893393 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:07.895644 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.895647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.895649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.895920 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.896167 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:07.896170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:07.896172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:07.896253 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:07.896615 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:07.896618 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:07.896715 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 15:21:07.896822 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 15:21:07.896955 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 15:21:07.897374 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.886993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.9.242.189:43205)
0927 15:21:07.887621 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:07.887627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:07.887637 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:21:07.887662 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:07.887664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:07.887712 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:07.887774 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:07.888255 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.888958 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.890158 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.890638 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.890738 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.893001 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:07.893010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:07.893023 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:07.893042 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:07.895018 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.895478 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.895484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.895486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.895576 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.895820 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.895823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.895825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.896103 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:07.896253 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:07.897081 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 15:21:07.897217 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":83,"threads_started":1}
I20260927 15:21:07.897750 531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.886348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.9.242.189:43205)
0927 15:21:07.886710 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:07.886721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:07.886743 (+ 22us) server_negotiation.cc:407] Connection header received
0927 15:21:07.886777 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:07.886781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:07.886953 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:07.887129 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:07.887502 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.888719 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.890495 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:07.890883 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:07.891018 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:07.893556 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:07.893559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:07.893562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:07.893585 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:07.895553 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.896009 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.896012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.896014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.896082 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:07.896329 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:07.896332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:07.896333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:07.896529 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:07.896612 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:07.897431 (+ 819us) server_negotiation.cc:299] Negotiation successful
0927 15:21:07.897540 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":235,"thread_start_us":153,"threads_started":1}
I20260927 15:21:07.899322 451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:07.899411 526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:07.900022 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.899796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50202)
0927 15:21:07.899904 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.899971 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 15:21:07.900214 474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
I20260927 15:21:07.900346 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.900065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50212)
0927 15:21:07.900224 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.900272 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 15:21:07.901017 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.900831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50234)
0927 15:21:07.900928 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.900972 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 15:21:07.901094 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.900819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50224)
0927 15:21:07.900988 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.901033 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260927 15:21:07.901181 526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:34765: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
W20260927 15:21:07.901270 451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:34765: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
I20260927 15:21:07.932426 451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } attempt: 1
I20260927 15:21:07.933314 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.933003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50244)
0927 15:21:07.933155 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.933228 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:21:07.933974 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.933793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50248)
0927 15:21:07.933877 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.933922 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 15:21:07.934093 451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:34765: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
I20260927 15:21:07.950441 526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } attempt: 1
I20260927 15:21:07.951309 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.950994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50250)
0927 15:21:07.951132 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.951202 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 15:21:07.952067 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:07.951862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50262)
0927 15:21:07.951964 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:07.952011 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 15:21:07.952214 526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:34765: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
W20260927 15:21:08.006724 533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 15:21:08.007028 533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 15:21:08.007053 533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 15:21:08.007125 533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 15:21:08.009259 451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } attempt: 2
I20260927 15:21:08.010133 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.009823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50266)
0927 15:21:08.009978 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.010044 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 15:21:08.010795 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.010592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50268)
0927 15:21:08.010687 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.010737 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 15:21:08.010929 451 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:34765: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
W20260927 15:21:08.011761 533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 15:21:08.011811 533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 15:21:08.011830 533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 15:21:08.011868 533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 15:21:08.011898 533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 15:21:08.017431 533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.242.148:34659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.242.190:33457,127.9.242.189:43205,127.9.242.188:34765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/krb5kdc/kudu_127.9.242.188.keytab
--principal=kudu/127.9.242.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.242.188:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.242.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 14:27:13 UTC on ccycloud.quasar-psqrth.root.comops.site
build id 4676138
I20260927 15:21:08.018568 533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 15:21:08.019658 533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 15:21:08.020378 526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } attempt: 2
I20260927 15:21:08.021170 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.020882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50280)
0927 15:21:08.021027 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.021086 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 15:21:08.021854 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.021633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50284)
0927 15:21:08.021748 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.021795 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 15:21:08.021996 526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.242.188:34765: Network error: Client connection negotiation failed: client connection to 127.9.242.188:34765: connect: Connection refused (error 111)
W20260927 15:21:08.028230 539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:08.028177 542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 15:21:08.028192 540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 15:21:08.029037 533 server_base.cc:1034] running on GCE node
I20260927 15:21:08.029605 533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 15:21:08.030728 533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 15:21:08.031890 533 hybrid_clock.cc:648] HybridClock initialized: now 1790522468031862 us; error 39 us; skew 500 ppm
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522468, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 15:21:08.034544 533 init.cc:377] Logged in from keytab as kudu/127.9.242.188@KRBTEST.COM (short username kudu)
I20260927 15:21:08.035410 533 webserver.cc:460] Webserver started at http://127.9.242.188:35823/ using document root <none> and password file <none>
I20260927 15:21:08.035941 533 fs_manager.cc:359] Metadata directory not provided
I20260927 15:21:08.036003 533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 15:21:08.036190 533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 15:21:08.038290 533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data/instance:
uuid: "59aa7d8c9184444784f2234de6413dbe"
format_stamp: "Formatted at 2026-09-27 15:21:08 on dist-test-slave-m4lt"
server_key: "a6010bdf5dad679d60cf77759811c432"
server_key_iv: "a4c796bf37cc638b39b9df9447365784"
server_key_version: "encryptionkey@0"
I20260927 15:21:08.038793 533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/wal/instance:
uuid: "59aa7d8c9184444784f2234de6413dbe"
format_stamp: "Formatted at 2026-09-27 15:21:08 on dist-test-slave-m4lt"
server_key: "a6010bdf5dad679d60cf77759811c432"
server_key_iv: "a4c796bf37cc638b39b9df9447365784"
server_key_version: "encryptionkey@0"
I20260927 15:21:08.042328 533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 15:21:08.044646 549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 15:21:08.045902 533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 15:21:08.046012 533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data,/tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/wal
uuid: "59aa7d8c9184444784f2234de6413dbe"
format_stamp: "Formatted at 2026-09-27 15:21:08 on dist-test-slave-m4lt"
server_key: "a6010bdf5dad679d60cf77759811c432"
server_key_iv: "a4c796bf37cc638b39b9df9447365784"
server_key_version: "encryptionkey@0"
I20260927 15:21:08.046185 533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 15:21:08.066800 533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 15:21:08.069602 533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 15:21:08.069806 533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 15:21:08.076635 533 rpc_server.cc:307] RPC server started. Bound to: 127.9.242.188:34765
I20260927 15:21:08.076895 601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.242.188:34765 every 8 connection(s)
I20260927 15:21:08.077589 533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/data/info.pb
I20260927 15:21:08.080035 10186 external_mini_cluster.cc:1388] Started /tmp/dist-test-task93l8wz/build/debug/bin/kudu as pid 533
I20260927 15:21:08.080173 10186 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task93l8wz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790522194092531-10186-0/minicluster-data/master-2/wal/instance
I20260927 15:21:08.080358 602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 15:21:08.080447 10186 external_mini_cluster.cc:1428] Setting key 8c2b21f577874db74ae55d5fb23bee18
I20260927 15:21:08.085117 602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522468, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:08.095827 605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.081757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.9.242.188:34765)
0927 15:21:08.082126 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.082140 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.082176 (+ 36us) server_negotiation.cc:407] Connection header received
0927 15:21:08.082958 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.082976 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.083335 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.083620 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.084533 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.085602 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.086763 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.087529 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.087673 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.090880 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.090885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.090895 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.090925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.092678 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.093149 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.093155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.093162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.093301 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.093589 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.093591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.093592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.094039 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.094285 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.094524 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.094816 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
I20260927 15:21:08.096539 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.087001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.190:33457 (local address 127.0.0.1:53202)
0927 15:21:08.087425 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.087466 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:08.087610 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:08.087889 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:08.087894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:08.088345 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:08.088486 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.088492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.089583 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.089586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.090614 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.090626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.091089 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.091092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.091225 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.091682 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:08.091693 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:08.093170 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:08.094919 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.094926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.094931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.095173 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.095392 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.095395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.095397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.095480 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.096039 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:08.096045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:08.096160 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 15:21:08.096285 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 15:21:08.096419 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":175,"threads_started":1}
I20260927 15:21:08.096648 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.087052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.9.242.190:33457)
0927 15:21:08.087162 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.087166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.087546 (+ 380us) server_negotiation.cc:407] Connection header received
0927 15:21:08.087704 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.087707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.087766 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.087843 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.088626 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.089419 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.090750 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.090966 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.091068 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.093317 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.093320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.093322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.093351 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.094827 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.095255 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.095258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.095259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.095326 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.095595 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.095598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.095599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.095831 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.095920 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.096374 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.096498 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:21:08.097589 602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522468, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.190@KRBTEST.COM
I20260927 15:21:08.106370 531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.098256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.9.242.189:43205)
0927 15:21:08.098399 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.098403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.098417 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:21:08.098460 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.098463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.098511 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.098591 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.099206 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.099782 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.100518 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.100769 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.100883 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.102847 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.102850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.102852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.102885 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.104737 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.105131 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.105134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.105136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.105220 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.105477 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.105481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.105482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.105766 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.105859 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.106083 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.106199 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 15:21:08.106389 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.098112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.189:43205 (local address 127.0.0.1:36200)
0927 15:21:08.098244 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.098259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:08.098336 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:08.098862 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:08.098865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:08.099000 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:08.099097 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.099102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.099883 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.099886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.100408 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.100415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.100868 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.100872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.100966 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.101405 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:08.101420 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:08.102684 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:08.104845 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.104847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.104849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.105033 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.105302 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.105305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.105306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.105380 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.105854 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:08.105857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:08.105942 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 15:21:08.106025 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 15:21:08.106250 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 15:21:08.106703 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.098531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.9.242.190:33457)
0927 15:21:08.098644 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.098647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.098657 (+ 10us) server_negotiation.cc:407] Connection header received
0927 15:21:08.098685 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.098687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.098719 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.098775 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.099354 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.099847 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.100582 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.100793 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.100922 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.103504 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.103508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.103510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.103537 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.105023 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.105585 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.105588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.105590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.105703 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.105932 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.105935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.105937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.106106 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.106309 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.106424 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.106531 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 15:21:08.107345 602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } has no permanent_uuid. Determining permanent_uuid...
I20260927 15:21:08.110106 451 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } attempt: 3
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522468, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.188@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.242.189@KRBTEST.COM
I20260927 15:21:08.113137 526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } attempt: 3
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.190@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:21:08.116392 606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.107791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50308)
0927 15:21:08.108003 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.108015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:08.108090 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:08.108329 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:08.108332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:08.108475 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:08.108545 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.108550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.109323 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.109326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.110013 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.110023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.110664 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.110667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.110737 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.111164 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:08.111178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:08.112471 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:08.114815 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.114818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.114820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.115242 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.115487 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.115489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.115491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.115570 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.115954 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:08.115956 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:08.116029 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 15:21:08.116121 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 15:21:08.116234 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260927 15:21:08.116863 605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.107912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.9.242.188:34765)
0927 15:21:08.108004 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.108009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.108053 (+ 44us) server_negotiation.cc:407] Connection header received
0927 15:21:08.108194 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.108197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.108240 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.108303 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.108644 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.109227 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.110125 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.110470 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.110570 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.112625 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.112629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.112632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.112659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.114732 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.115334 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.115337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.115338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.115404 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.115651 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.115654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.115655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.115841 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.115923 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.116622 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.116719 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 27 15:21:08 dist-test-slave-m4lt krb5kdc[370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790522467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.242.189@KRBTEST.COM for kudu/127.9.242.188@KRBTEST.COM
I20260927 15:21:08.120229 453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.110447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50312)
0927 15:21:08.110603 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.110617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:08.110687 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:08.111688 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:08.111692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:08.111858 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:08.111959 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.111966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.113311 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.113315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.114150 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.114162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.114790 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.114794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.114874 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.115270 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:08.115283 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:08.116653 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:08.118792 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.118795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.118797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.118979 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.119287 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.119290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.119292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.119368 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.119765 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:08.119768 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:08.119861 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 15:21:08.119964 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 15:21:08.120084 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 15:21:08.120340 607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.110502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.9.242.188:34765)
0927 15:21:08.111455 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.111464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.111476 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:21:08.111506 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.111509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.111551 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.111640 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.112087 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.113184 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.114296 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.114660 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.115077 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.117245 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.117248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.117249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.117269 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.118701 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.119072 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.119075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.119077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.119161 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.119449 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.119452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.119454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.119644 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.119765 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.120028 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.120176 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":127,"threads_started":1}
I20260927 15:21:08.120362 602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: Bootstrap starting.
I20260927 15:21:08.121804 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.113803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.9.242.188:34765)
0927 15:21:08.114022 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.114030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.114042 (+ 12us) server_negotiation.cc:407] Connection header received
0927 15:21:08.114072 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.114075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.114114 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.114308 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.114655 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.115392 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.116349 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.116668 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.117035 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.118651 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.118655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.118657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.118695 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.120383 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.120779 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.120781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.120782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.120841 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.121101 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.121103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.121104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.121233 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.121296 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.121529 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.121634 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"spinlock_wait_cycles":13056,"thread_start_us":85,"threads_started":1}
I20260927 15:21:08.121809 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.113456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.242.188:34765 (local address 127.0.0.1:50322)
0927 15:21:08.113610 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 15:21:08.113623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 15:21:08.113792 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 15:21:08.114318 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 15:21:08.114321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 15:21:08.114482 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 15:21:08.114551 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.114556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.115493 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.115496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.116227 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 15:21:08.116235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.116782 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.116785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 15:21:08.116865 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.117234 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 15:21:08.117247 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 15:21:08.118513 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 15:21:08.120496 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.120499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.120501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.120689 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.120927 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 15:21:08.120930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 15:21:08.120932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 15:21:08.121023 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 15:21:08.121308 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 15:21:08.121311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 15:21:08.121396 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 15:21:08.121502 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 15:21:08.121637 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 15:21:08.122442 531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.107869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.9.242.189:43205)
0927 15:21:08.107972 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.107975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.108108 (+ 133us) server_negotiation.cc:407] Connection header received
0927 15:21:08.108208 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.108210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.108248 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.108306 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.108893 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.109502 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.110528 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.110836 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.110904 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.113367 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.113371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.113374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.113397 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.115332 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.115784 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.115787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.115789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.115865 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.116114 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.116116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.116118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.116310 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.116475 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.122216 (+ 5741us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.122326 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 15:21:08.123406 602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:08.124420 602 log.cc:824] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:08.124681 451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9: Bootstrap starting.
I20260927 15:21:08.125401 10186 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 15:21:08.126554 602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: No bootstrap required, opened a new log
I20260927 15:21:08.127265 451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:08.126473 425 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2b7901c47f364235bf3af953e448eab9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53216
I20260927 15:21:08.128095 451 log.cc:824] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:08.130204 451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9: No bootstrap required, opened a new log
I20260927 15:21:08.130673 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: Bootstrap starting.
I20260927 15:21:08.131011 602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.131254 602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59aa7d8c9184444784f2234de6413dbe, State: Initialized, Role: FOLLOWER
I20260927 15:21:08.131573 602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.132131 611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.132416 611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:08.132866 602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:08.132941 451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.133129 451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7901c47f364235bf3af953e448eab9, State: Initialized, Role: FOLLOWER
I20260927 15:21:08.133184 526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: Neither blocks nor log segments found. Creating new log.
I20260927 15:21:08.133419 451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.134051 526 log.cc:824] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: Log is configured to *not* fsync() on all Append() calls
I20260927 15:21:08.134024 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.134337 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:08.134713 451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:08.134749 425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 15:21:08.134859 425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:08.135989 425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.136869 425 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [CANDIDATE]: Term 1 election: Requested vote from peers 410b17e355d24deaa35e59fa2d588d44 (127.9.242.189:43205), 59aa7d8c9184444784f2234de6413dbe (127.9.242.188:34765)
I20260927 15:21:08.137705 576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b7901c47f364235bf3af953e448eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "59aa7d8c9184444784f2234de6413dbe"
I20260927 15:21:08.137904 576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:08.139259 576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7901c47f364235bf3af953e448eab9 in term 1.
I20260927 15:21:08.136480 533 master_runner.cc:186] Error getting master registration for 127.9.242.190:33457: OK, instance_id { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" instance_seqno: 1790522467641798 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260927 15:21:08.139791 404 tablet.cc:2369] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 15:21:08.139856 533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:21:08.140403 399 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7901c47f364235bf3af953e448eab9, 59aa7d8c9184444784f2234de6413dbe; no voters:
I20260927 15:21:08.140664 613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 15:21:08.140875 624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:21:08.140941 624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:21:08.140918 613 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7901c47f364235bf3af953e448eab9, State: Running, Role: LEADER
I20260927 15:21:08.141224 613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.139107 383 master_runner.cc:186] Error getting master registration for 127.9.242.189:43205: OK, instance_id { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" instance_seqno: 1790522467859245 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 15:21:08.142179 383 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 15:21:08.142937 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7901c47f364235bf3af953e448eab9. Latest consensus state: current_term: 1 leader_uuid: "2b7901c47f364235bf3af953e448eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.143070 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:08.143409 643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 15:21:08.146363 643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 15:21:08.147938 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: No bootstrap required, opened a new log
I20260927 15:21:08.152118 526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.152354 526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410b17e355d24deaa35e59fa2d588d44, State: Initialized, Role: FOLLOWER
I20260927 15:21:08.152683 526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } }
I20260927 15:21:08.154095 526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 15:21:08.154152 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2b7901c47f364235bf3af953e448eab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "410b17e355d24deaa35e59fa2d588d44"
I20260927 15:21:08.154372 499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 15:21:08.155704 499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7901c47f364235bf3af953e448eab9 in term 1.
I20260927 15:21:08.161820 649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.161947 649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:08.165464 576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [term 1 FOLLOWER]: Refusing update from remote peer 2b7901c47f364235bf3af953e448eab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:08.165791 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.143892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.9.242.188:34765)
0927 15:21:08.144010 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.144014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.148451 (+ 4437us) server_negotiation.cc:407] Connection header received
0927 15:21:08.148679 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.148682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.148733 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.148782 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.149783 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.150547 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.151921 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.152257 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.152354 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.156470 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.156474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.156477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.156507 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.158228 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.163577 (+ 5349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.163581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.163583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.163675 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.163964 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.163967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.163968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.164214 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.164338 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.165483 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.165633 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:21:08.166108 660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 15:21:08.166195 660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 15:21:08.166193 641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:08.168006 499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [term 1 FOLLOWER]: Refusing update from remote peer 2b7901c47f364235bf3af953e448eab9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 15:21:08.168226 531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.142562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.9.242.189:43205)
0927 15:21:08.142698 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.142703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.144688 (+ 1985us) server_negotiation.cc:407] Connection header received
0927 15:21:08.146105 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.146108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.146158 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.146211 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.147096 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.147777 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.148802 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.149120 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.149191 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.159216 (+ 10025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.159220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.159223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.159252 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.160905 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.164858 (+ 3953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.164862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.164864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.164953 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.166315 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.166318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.166320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.166537 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.166645 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.167969 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.168096 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 15:21:08.168497 641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 15:21:08.168915 455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:08.143039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.9.242.190:33457)
0927 15:21:08.143158 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.143162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.144484 (+ 1322us) server_negotiation.cc:407] Connection header received
0927 15:21:08.144879 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.144883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.144931 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.144981 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.145993 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.146701 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.148472 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:08.148797 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:08.148864 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:08.161454 (+ 12590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 15:21:08.161458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 15:21:08.161461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 15:21:08.161490 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 15:21:08.163104 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.164654 (+ 1550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.164657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.164659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.164746 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 15:21:08.167250 (+ 2504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 15:21:08.167253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 15:21:08.167255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 15:21:08.167471 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 15:21:08.167574 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:08.168668 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0927 15:21:08.168783 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 15:21:08.170755 611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7901c47f364235bf3af953e448eab9. Latest consensus state: current_term: 1 leader_uuid: "2b7901c47f364235bf3af953e448eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.170866 611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:08.172385 649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7901c47f364235bf3af953e448eab9. Latest consensus state: current_term: 1 leader_uuid: "2b7901c47f364235bf3af953e448eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.172502 649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:08.173892 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b7901c47f364235bf3af953e448eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.173991 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:08.174206 611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b7901c47f364235bf3af953e448eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.174330 611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:08.175594 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b7901c47f364235bf3af953e448eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.175701 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9 [sys.catalog]: This master's current role is: LEADER
I20260927 15:21:08.176606 649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b7901c47f364235bf3af953e448eab9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b7901c47f364235bf3af953e448eab9" member_type: VOTER last_known_addr { host: "127.9.242.190" port: 33457 } } peers { permanent_uuid: "410b17e355d24deaa35e59fa2d588d44" member_type: VOTER last_known_addr { host: "127.9.242.189" port: 43205 } } peers { permanent_uuid: "59aa7d8c9184444784f2234de6413dbe" member_type: VOTER last_known_addr { host: "127.9.242.188" port: 34765 } } }
I20260927 15:21:08.176710 649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 15:21:08.188591 662 mvcc.cc:205] Tried to move back new op lower bound from 7333980029597704192 to 7333980029509877760. Current Snapshot: MvccSnapshot[applied={T|T < 7333980029597704192}]
I20260927 15:21:08.190426 643 catalog_manager.cc:1347] Generated new cluster ID: 8af7c9c59db745c19acd6353e67380b4
I20260927 15:21:08.190502 643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 15:21:08.207461 643 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 15:21:08.208871 643 catalog_manager.cc:1504] Loading token signing keys...
I20260927 15:21:08.226679 643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b7901c47f364235bf3af953e448eab9: Generated new TSK 0
I20260927 15:21:08.227680 643 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 15:21:08.264923 455 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.261790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.9.242.190:33457)
0927 15:21:08.261919 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.261924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.263799 (+ 1875us) server_negotiation.cc:407] Connection header received
0927 15:21:08.264018 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.264022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.264090 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.264143 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.264777 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53232: BlockingRecv error: recv got EOF from 127.0.0.1:53232 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 15:21:08.266384 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.262666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.9.242.189:43205)
0927 15:21:08.262780 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.262784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.265404 (+ 2620us) server_negotiation.cc:407] Connection header received
0927 15:21:08.265585 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.265589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.265637 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.265734 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.266262 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36222: BlockingRecv error: recv got EOF from 127.0.0.1:36222 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 15:21:08.267705 608 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.263305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.9.242.188:34765)
0927 15:21:08.263413 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.263417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.266848 (+ 3431us) server_negotiation.cc:407] Connection header received
0927 15:21:08.267024 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.267027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.267087 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.267135 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.267586 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50332: BlockingRecv error: recv got EOF from 127.0.0.1:50332 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 15:21:08.272076 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.270470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.9.242.189:43205)
0927 15:21:08.270581 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.270584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.271129 (+ 545us) server_negotiation.cc:407] Connection header received
0927 15:21:08.271348 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.271351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.271424 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.271475 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.271948 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36236: BlockingRecv error: recv got EOF from 127.0.0.1:36236 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 15:21:08.276767 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.275392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.9.242.189:43205)
0927 15:21:08.275488 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.275492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.275927 (+ 435us) server_negotiation.cc:407] Connection header received
0927 15:21:08.276112 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.276115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.276163 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.276211 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.276670 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36240: BlockingRecv error: recv got EOF from 127.0.0.1:36240 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 15:21:08.282374 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.280835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.9.242.189:43205)
0927 15:21:08.280942 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.280946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.281424 (+ 478us) server_negotiation.cc:407] Connection header received
0927 15:21:08.281605 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.281608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.281684 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.281734 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.282220 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36252: BlockingRecv error: recv got EOF from 127.0.0.1:36252 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 15:21:08.290127 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.288507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.9.242.189:43205)
0927 15:21:08.288649 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.288652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.289161 (+ 509us) server_negotiation.cc:407] Connection header received
0927 15:21:08.289356 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.289360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.289409 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.289458 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.290007 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36262: BlockingRecv error: recv got EOF from 127.0.0.1:36262 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 15:21:08.302485 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.300752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.9.242.189:43205)
0927 15:21:08.300877 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.300882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.301444 (+ 562us) server_negotiation.cc:407] Connection header received
0927 15:21:08.301690 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.301694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.301753 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.301804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.302367 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36272: BlockingRecv error: recv got EOF from 127.0.0.1:36272 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 15:21:08.322906 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.321218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.9.242.189:43205)
0927 15:21:08.321341 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.321345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.321934 (+ 589us) server_negotiation.cc:407] Connection header received
0927 15:21:08.322126 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.322130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.322185 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.322234 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.322770 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36286: BlockingRecv error: recv got EOF from 127.0.0.1:36286 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 15:21:08.358747 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.357615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.9.242.189:43205)
0927 15:21:08.357751 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.357757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.357923 (+ 166us) server_negotiation.cc:407] Connection header received
0927 15:21:08.358038 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.358041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.358097 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.358190 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.358586 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36296: BlockingRecv error: recv got EOF from 127.0.0.1:36296 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 15:21:08.425786 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.424650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.9.242.189:43205)
0927 15:21:08.424804 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.424808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.424959 (+ 151us) server_negotiation.cc:407] Connection header received
0927 15:21:08.425089 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.425093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.425151 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.425240 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.425617 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36312: BlockingRecv error: recv got EOF from 127.0.0.1:36312 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 15:21:08.557152 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.555878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.9.242.189:43205)
0927 15:21:08.556028 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.556033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.556276 (+ 243us) server_negotiation.cc:407] Connection header received
0927 15:21:08.556427 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.556430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.556477 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.556545 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.557031 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36318: BlockingRecv error: recv got EOF from 127.0.0.1:36318 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 15:21:08.816346 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 15:21:08.815352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.9.242.189:43205)
0927 15:21:08.815479 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:08.815484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:08.815499 (+ 15us) server_negotiation.cc:407] Connection header received
0927 15:21:08.815644 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:08.815647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:08.815701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:08.815772 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 15:21:08.816227 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36324: BlockingRecv error: recv got EOF from 127.0.0.1:36324 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 15:21:09.142344 624 catalog_manager.cc:1259] Loaded cluster ID: 8af7c9c59db745c19acd6353e67380b4
I20260927 15:21:09.142426 624 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: loading cluster ID for follower catalog manager: success
I20260927 15:21:09.144032 624 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: acquiring CA information for follower catalog manager: success
I20260927 15:21:09.145154 624 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59aa7d8c9184444784f2234de6413dbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:21:09.167438 660 catalog_manager.cc:1259] Loaded cluster ID: 8af7c9c59db745c19acd6353e67380b4
I20260927 15:21:09.167522 660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: loading cluster ID for follower catalog manager: success
I20260927 15:21:09.169195 660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: acquiring CA information for follower catalog manager: success
I20260927 15:21:09.170375 660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 410b17e355d24deaa35e59fa2d588d44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 15:21:09.334533 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:09.330449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.9.242.189:43205)
0927 15:21:09.330720 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:09.330724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:09.330763 (+ 39us) server_negotiation.cc:407] Connection header received
0927 15:21:09.330814 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:09.330817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:09.330937 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:09.331034 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:09.331412 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.331961 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.332842 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.333071 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.333189 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:09.333500 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:09.334010 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:09.334129 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:09.334219 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 15:21:09.334363 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260927 15:21:09.341152 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:09.337624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.9.242.188:34765)
0927 15:21:09.337911 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:09.337914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:09.337928 (+ 14us) server_negotiation.cc:407] Connection header received
0927 15:21:09.338005 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:09.338007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:09.338129 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:09.338235 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:09.338407 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.338915 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.339767 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.340000 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.340074 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:09.340391 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:09.340838 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:09.340942 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:09.340971 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 15:21:09.341029 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
I20260927 15:21:09.348197 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:09.344774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.9.242.188:34765)
0927 15:21:09.344881 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:09.344885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:09.344910 (+ 25us) server_negotiation.cc:407] Connection header received
0927 15:21:09.344999 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:09.345002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:09.345046 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:09.345112 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:09.345463 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.346215 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.347103 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.347444 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.347545 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:09.347766 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:09.347845 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:09.347901 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:09.348012 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 15:21:09.348071 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 15:21:09.348580 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:09.344060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.9.242.189:43205)
0927 15:21:09.344193 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:09.344198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:09.344487 (+ 289us) server_negotiation.cc:407] Connection header received
0927 15:21:09.344677 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:09.344681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:09.344739 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:09.344794 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:09.345143 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.345821 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.346964 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.347260 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.347347 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:09.347583 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:09.347681 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:09.347725 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:09.348402 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 15:21:09.348462 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 15:21:09.348896 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 15:21:09.344027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.9.242.190:33457)
0927 15:21:09.344274 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 15:21:09.344279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 15:21:09.344471 (+ 192us) server_negotiation.cc:407] Connection header received
0927 15:21:09.344636 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 15:21:09.344640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 15:21:09.344851 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 15:21:09.344965 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 15:21:09.345229 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.346002 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.347142 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 15:21:09.347480 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 15:21:09.347561 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 15:21:09.347948 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 15:21:09.348254 (+ 306us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 15:21:09.348349 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 15:21:09.348689 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 15:21:09.348758 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260927 15:21:09.351753 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 383
I20260927 15:21:09.357357 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 458
I20260927 15:21:09.362501 10186 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task93l8wz/build/debug/bin/kudu with pid 533
2026-09-27T15:21:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2971 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5997 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275263 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST